[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:58 dist-test-slave-wp6g krb5kdc[11248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:58 dist-test-slave-wp6g krb5kdc[11248](info): set up 2 sockets
Jun 20 18:38:58 dist-test-slave-wp6g krb5kdc[11248](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 18:38:59.747308 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:38:59 dist-test-slave-wp6g krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:38:59.871376 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:46581
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:46581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:59.954639 11261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:59.954855 11261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:59.954879 11261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:59.956086 11261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:59.956131 11261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:59.956146 11261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:59.956159 11261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:59.956207 11261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:59.957722 11261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:46581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:46581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:59.957990 11261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:59.958236 11261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:59.963896 11266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:59.963891 11267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:59.963891 11269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:59.963924 11261 server_base.cc:1034] running on GCE node
I20260620 18:38:59.964356 11261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:59.964696 11261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:59.965874 11261 hybrid_clock.cc:648] HybridClock initialized: now 1781980739965851 us; error 41 us; skew 500 ppm
Jun 20 18:38:59 dist-test-slave-wp6g krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980739, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:59.968618 11261 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:38:59.969085 11261 webserver.cc:460] Webserver started at http://127.10.250.126:33459/ using document root <none> and password file <none>
I20260620 18:38:59.969345 11261 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:59.969417 11261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:59.969566 11261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:59.970752 11261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "ff0622305f394acebf626297492a71b1"
format_stamp: "Formatted at 2026-06-20 18:38:59 on dist-test-slave-wp6g"
server_key: "40532b46cef1b6b5f3a511f5bf76a6b8"
server_key_iv: "e6e17e637a340fc247d71ee67bf22035"
server_key_version: "encryptionkey@0"
I20260620 18:38:59.971117 11261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "ff0622305f394acebf626297492a71b1"
format_stamp: "Formatted at 2026-06-20 18:38:59 on dist-test-slave-wp6g"
server_key: "40532b46cef1b6b5f3a511f5bf76a6b8"
server_key_iv: "e6e17e637a340fc247d71ee67bf22035"
server_key_version: "encryptionkey@0"
I20260620 18:38:59.972579 11261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:59.973443 11276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:59.973642 11261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:59.973754 11261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "ff0622305f394acebf626297492a71b1"
format_stamp: "Formatted at 2026-06-20 18:38:59 on dist-test-slave-wp6g"
server_key: "40532b46cef1b6b5f3a511f5bf76a6b8"
server_key_iv: "e6e17e637a340fc247d71ee67bf22035"
server_key_version: "encryptionkey@0"
I20260620 18:38:59.973848 11261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:59.986232 11261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:59.988768 11261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:59.988945 11261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:59.993003 11261 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:46581
I20260620 18:38:59.993057 11328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:46581 every 8 connection(s)
I20260620 18:38:59.993439 11261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:59.994395 11329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:59.996475 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 11261
I20260620 18:38:59.996580 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:38:59.997295 11329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1: Bootstrap starting.
I20260620 18:38:59.997383 11241 external_mini_cluster.cc:1428] Setting key 6a79016ce4db9c9fd98f3bdf955c8c92
I20260620 18:38:59.998045 11329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:59.998425 11329 log.cc:824] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:59.999246 11329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1: No bootstrap required, opened a new log
I20260620 18:39:00.002252 11329 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0622305f394acebf626297492a71b1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46581 } }
I20260620 18:39:00.002457 11329 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:00.002539 11329 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff0622305f394acebf626297492a71b1, State: Initialized, Role: FOLLOWER
I20260620 18:39:00.002743 11329 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0622305f394acebf626297492a71b1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46581 } }
I20260620 18:39:00.002847 11329 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:00.002895 11329 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:00.002948 11329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:00.004215 11329 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0622305f394acebf626297492a71b1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46581 } }
I20260620 18:39:00.004366 11329 leader_election.cc:302] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff0622305f394acebf626297492a71b1; no voters:
I20260620 18:39:00.004550 11329 leader_election.cc:288] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:00.004719 11334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:00.004843 11329 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:00.004925 11334 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [term 1 LEADER]: Becoming Leader. State: Replica: ff0622305f394acebf626297492a71b1, State: Running, Role: LEADER
I20260620 18:39:00.005084 11334 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0622305f394acebf626297492a71b1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46581 } }
I20260620 18:39:00.005450 11334 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff0622305f394acebf626297492a71b1. Latest consensus state: current_term: 1 leader_uuid: "ff0622305f394acebf626297492a71b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0622305f394acebf626297492a71b1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46581 } } }
I20260620 18:39:00.005518 11334 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:00.006506 11336 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff0622305f394acebf626297492a71b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0622305f394acebf626297492a71b1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46581 } } }
I20260620 18:39:00.006641 11336 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1 [sys.catalog]: This master's current role is: LEADER
W20260620 18:39:00.007259 11349 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:39:00.007331 11349 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:39:00.007593 11347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:00.007809 11347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:00.009517 11347 catalog_manager.cc:1347] Generated new cluster ID: 80d93bd8de9248ae9c342890cb048d5d
I20260620 18:39:00.009618 11347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:39:00 dist-test-slave-wp6g krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:00.019883 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.998384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58396 (local address 127.10.250.126:46581)
0620 18:38:59.998663 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:59.998683 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:59.998832 (+ 149us) server_negotiation.cc:407] Connection header received
0620 18:39:00.001616 (+ 2784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:00.001629 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:00.001717 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:00.002035 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:00.004075 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.005928 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.007770 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.008395 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.008532 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.013748 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:00.013755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:00.013755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:00.013772 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:00.016908 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.017557 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.017563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.017563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.017702 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.018024 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.018025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.018026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.018606 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:00.018788 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:00.019154 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 18:39:00.019330 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":103,"threads_started":1}
I20260620 18:39:00.021003 11347 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:00.021533 11347 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:00.024457 11347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff0622305f394acebf626297492a71b1: Generated new TSK 0
I20260620 18:39:00.024648 11347 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:39:00.055655 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:46581
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:00.142030 11355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:00.142252 11355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:00.142278 11355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:00.143424 11355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:00.143463 11355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:00.143489 11355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:00.144899 11355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:46581
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:00.145102 11355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:00.145299 11355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:00.150874 11363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:00.150876 11361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:00.150876 11360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:00.151106 11355 server_base.cc:1034] running on GCE node
I20260620 18:39:00.151319 11355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:00.151587 11355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:00.152779 11355 hybrid_clock.cc:648] HybridClock initialized: now 1781980740152762 us; error 28 us; skew 500 ppm
Jun 20 18:39:00 dist-test-slave-wp6g krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:00.155447 11355 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:39:00.155896 11355 webserver.cc:460] Webserver started at http://127.10.250.65:34733/ using document root <none> and password file <none>
I20260620 18:39:00.156143 11355 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:00.156211 11355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:00.156364 11355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:00.157452 11355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "bae89bc8c8c8445483106f73945283ca"
format_stamp: "Formatted at 2026-06-20 18:39:00 on dist-test-slave-wp6g"
server_key: "4309c92b695740d02390ec29f0608114"
server_key_iv: "8182d95d122222cf84881eb09787039f"
server_key_version: "encryptionkey@0"
I20260620 18:39:00.157828 11355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "bae89bc8c8c8445483106f73945283ca"
format_stamp: "Formatted at 2026-06-20 18:39:00 on dist-test-slave-wp6g"
server_key: "4309c92b695740d02390ec29f0608114"
server_key_iv: "8182d95d122222cf84881eb09787039f"
server_key_version: "encryptionkey@0"
I20260620 18:39:00.159279 11355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:39:00.160188 11370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:00.160382 11355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:00.160468 11355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "bae89bc8c8c8445483106f73945283ca"
format_stamp: "Formatted at 2026-06-20 18:39:00 on dist-test-slave-wp6g"
server_key: "4309c92b695740d02390ec29f0608114"
server_key_iv: "8182d95d122222cf84881eb09787039f"
server_key_version: "encryptionkey@0"
I20260620 18:39:00.160524 11355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:00.169477 11355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:00.171799 11355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:00.171921 11355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:00.172115 11355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:00.172478 11355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:00.172514 11355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:00.172537 11355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:00.172550 11355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:00.178455 11355 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:35749
I20260620 18:39:00.178541 11483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:35749 every 8 connection(s)
I20260620 18:39:00.178905 11355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:00.180667 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 11355
I20260620 18:39:00.180750 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:00.180900 11241 external_mini_cluster.cc:1428] Setting key 6923e301437d6afa09bac603da4aab3e
Jun 20 18:39:00 dist-test-slave-wp6g krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
I20260620 18:39:00.188699 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.179655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:51707 (local address 127.10.250.126:46581)
0620 18:39:00.179716 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:00.179718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:00.179960 (+ 242us) server_negotiation.cc:407] Connection header received
0620 18:39:00.180423 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:00.180424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:00.180434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:00.180523 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:00.181534 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.182054 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.183224 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.183433 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.183519 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.185764 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:00.185765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:00.185766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:00.185775 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:00.187248 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.187672 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.187673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.187674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.187736 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.187991 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.187992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.187993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.188121 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:00.188202 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:00.188502 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 18:39:00.188603 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:39:00.188912 11486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.179665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46581 (local address 127.10.250.65:51707)
0620 18:39:00.179833 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:00.179870 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:00.180340 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:00.180540 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:00.180543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:00.181230 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:00.181431 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:00.181436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.182158 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.182159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:00.183109 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:00.183112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.183534 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.183535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:00.183680 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.184240 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:00.184249 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:00.185652 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:00.187336 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:00.187345 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:00.187348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:00.187578 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:00.187803 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:00.187804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:00.187804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:00.187918 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:00.188204 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:00.188212 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:00.188359 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 18:39:00.188468 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:39:00.188641 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260620 18:39:00.189558 11484 heartbeater.cc:347] Connected to a master server at 127.10.250.126:46581
I20260620 18:39:00.189725 11484 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:00.189945 11484 heartbeater.cc:511] Master 127.10.250.126:46581 requested a full tablet report, sending...
I20260620 18:39:00.190644 11293 ts_manager.cc:194] Registered new tserver with Master: bae89bc8c8c8445483106f73945283ca (127.10.250.65:35749)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
I20260620 18:39:00.191329 11293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:51707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:39:00.206027 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:46581
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:00.290199 11489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:00.290439 11489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:00.290468 11489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:00.291852 11489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:00.291909 11489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:00.291939 11489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:00.293514 11489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:46581
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:00.293804 11489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:00.294063 11489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:00.299232 11497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:00.299230 11495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:00.299284 11494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:00.299638 11489 server_base.cc:1034] running on GCE node
I20260620 18:39:00.299888 11489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:00.300094 11489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:00.301244 11489 hybrid_clock.cc:648] HybridClock initialized: now 1781980740301230 us; error 28 us; skew 500 ppm
Jun 20 18:39:00 dist-test-slave-wp6g krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:00.303862 11489 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:39:00.304287 11489 webserver.cc:460] Webserver started at http://127.10.250.66:35671/ using document root <none> and password file <none>
I20260620 18:39:00.304554 11489 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:00.304620 11489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:00.304726 11489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:00.305883 11489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "22b2897815ed49edb537a1cd3afa7835"
format_stamp: "Formatted at 2026-06-20 18:39:00 on dist-test-slave-wp6g"
server_key: "7f9d94fe5340786ac4a5b00711605f86"
server_key_iv: "23efcbb4fe3faa34640f2481a0776c2a"
server_key_version: "encryptionkey@0"
I20260620 18:39:00.306258 11489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "22b2897815ed49edb537a1cd3afa7835"
format_stamp: "Formatted at 2026-06-20 18:39:00 on dist-test-slave-wp6g"
server_key: "7f9d94fe5340786ac4a5b00711605f86"
server_key_iv: "23efcbb4fe3faa34640f2481a0776c2a"
server_key_version: "encryptionkey@0"
I20260620 18:39:00.307788 11489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:00.308634 11504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:00.308874 11489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:00.308981 11489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "22b2897815ed49edb537a1cd3afa7835"
format_stamp: "Formatted at 2026-06-20 18:39:00 on dist-test-slave-wp6g"
server_key: "7f9d94fe5340786ac4a5b00711605f86"
server_key_iv: "23efcbb4fe3faa34640f2481a0776c2a"
server_key_version: "encryptionkey@0"
I20260620 18:39:00.309057 11489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:00.333523 11489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:00.335908 11489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:00.336035 11489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:00.336231 11489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:00.336552 11489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:00.336585 11489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:00.336608 11489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:00.336644 11489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:00.342633 11489 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:39037
I20260620 18:39:00.342689 11617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:39037 every 8 connection(s)
I20260620 18:39:00.342995 11489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:39:00 dist-test-slave-wp6g krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:00.351617 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 11489
I20260620 18:39:00.351707 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:00.351897 11241 external_mini_cluster.cc:1428] Setting key 55b7bed4796a5240ee8f9a2d3b4a75ac
I20260620 18:39:00.353135 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.343748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:35849 (local address 127.10.250.126:46581)
0620 18:39:00.343822 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:00.343824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:00.344044 (+ 220us) server_negotiation.cc:407] Connection header received
0620 18:39:00.344598 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:00.344599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:00.344610 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:00.344669 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:00.345628 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.346136 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.347374 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.347573 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.347680 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.349972 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:00.349973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:00.349974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:00.349981 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:00.351369 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.351928 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.351930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.351930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.351998 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.352369 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.352370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.352370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.352518 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:00.352574 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:00.352942 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 18:39:00.353038 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 18:39:00.353390 11620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.343751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46581 (local address 127.10.250.66:35849)
0620 18:39:00.343923 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:00.343960 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:00.344484 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:00.344746 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:00.344750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:00.345322 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:00.345516 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:00.345521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.346252 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.346253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:00.347267 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:00.347270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.347689 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.347690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:00.347830 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.348334 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:00.348342 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:00.349859 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:00.351468 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:00.351476 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:00.351478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:00.351847 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:00.352102 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:00.352103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:00.352103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:00.352293 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:00.352606 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:00.352615 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:00.352788 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 18:39:00.352916 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 18:39:00.353141 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260620 18:39:00.353873 11618 heartbeater.cc:347] Connected to a master server at 127.10.250.126:46581
I20260620 18:39:00.353984 11618 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:00.354202 11618 heartbeater.cc:511] Master 127.10.250.126:46581 requested a full tablet report, sending...
I20260620 18:39:00.354672 11293 ts_manager.cc:194] Registered new tserver with Master: 22b2897815ed49edb537a1cd3afa7835 (127.10.250.66:39037)
I20260620 18:39:00.355204 11293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:35849
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:39:00.378850 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:46581
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:00.464226 11623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:00.464438 11623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:00.464464 11623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:00.465638 11623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:00.465690 11623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:00.465744 11623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:00.467391 11623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:46581
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:00.467658 11623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:00.467914 11623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:00.473150 11628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:00.473167 11629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:00.473289 11631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:00.473344 11623 server_base.cc:1034] running on GCE node
I20260620 18:39:00.473730 11623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:00.474009 11623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:00.475174 11623 hybrid_clock.cc:648] HybridClock initialized: now 1781980740475152 us; error 42 us; skew 500 ppm
Jun 20 18:39:00 dist-test-slave-wp6g krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:00.477659 11623 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:39:00.478227 11623 webserver.cc:460] Webserver started at http://127.10.250.67:41173/ using document root <none> and password file <none>
I20260620 18:39:00.478490 11623 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:00.478585 11623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:00.478729 11623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:00.479820 11623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "3a474d15be3c4965b932d2728dc5954e"
format_stamp: "Formatted at 2026-06-20 18:39:00 on dist-test-slave-wp6g"
server_key: "c9b6ef4837e0c5b997cfcfeb445f74e6"
server_key_iv: "fb194a461cbc4a3e19c357d09a5377be"
server_key_version: "encryptionkey@0"
I20260620 18:39:00.480198 11623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "3a474d15be3c4965b932d2728dc5954e"
format_stamp: "Formatted at 2026-06-20 18:39:00 on dist-test-slave-wp6g"
server_key: "c9b6ef4837e0c5b997cfcfeb445f74e6"
server_key_iv: "fb194a461cbc4a3e19c357d09a5377be"
server_key_version: "encryptionkey@0"
I20260620 18:39:00.481535 11623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:00.482321 11638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:00.482589 11623 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:00.482684 11623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "3a474d15be3c4965b932d2728dc5954e"
format_stamp: "Formatted at 2026-06-20 18:39:00 on dist-test-slave-wp6g"
server_key: "c9b6ef4837e0c5b997cfcfeb445f74e6"
server_key_iv: "fb194a461cbc4a3e19c357d09a5377be"
server_key_version: "encryptionkey@0"
I20260620 18:39:00.482774 11623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:00.509817 11623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:00.512005 11623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:00.512223 11623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:00.512481 11623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:00.512807 11623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:00.512840 11623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:00.512884 11623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:00.512912 11623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:00.518914 11623 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:33365
I20260620 18:39:00.518956 11751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:33365 every 8 connection(s)
I20260620 18:39:00.519317 11623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:00.523873 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 11623
I20260620 18:39:00.523972 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:00.524135 11241 external_mini_cluster.cc:1428] Setting key e39cc5621dcaef93bde5e5c16e755ecc
Jun 20 18:39:00 dist-test-slave-wp6g krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:00.529376 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.520112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:58867 (local address 127.10.250.126:46581)
0620 18:39:00.520184 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:00.520187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:00.520457 (+ 270us) server_negotiation.cc:407] Connection header received
0620 18:39:00.520970 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:00.520971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:00.520982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:00.521046 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:00.522003 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.522507 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.523592 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.523930 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.524011 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.526592 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:00.526594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:00.526594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:00.526602 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:00.528062 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.528467 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.528468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.528469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.528525 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.528768 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.528768 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.528768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.528887 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:00.528939 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:00.529239 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 18:39:00.529305 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:39:00.529633 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.520098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46581 (local address 127.10.250.67:58867)
0620 18:39:00.520281 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:00.520319 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:00.520857 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:00.521143 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:00.521146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:00.521724 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:00.521925 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:00.521930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.522666 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.522668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:00.523508 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:00.523511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.524065 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.524067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:00.524248 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.524897 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:00.524909 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:00.526453 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:00.528149 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:00.528162 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:00.528164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:00.528395 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:00.528587 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:00.528587 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:00.528588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:00.528713 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:00.528953 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:00.528957 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:00.529092 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 18:39:00.529208 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:39:00.529352 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260620 18:39:00.530227 11752 heartbeater.cc:347] Connected to a master server at 127.10.250.126:46581
I20260620 18:39:00.530337 11752 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:00.530591 11752 heartbeater.cc:511] Master 127.10.250.126:46581 requested a full tablet report, sending...
I20260620 18:39:00.531049 11293 ts_manager.cc:194] Registered new tserver with Master: 3a474d15be3c4965b932d2728dc5954e (127.10.250.67:33365)
I20260620 18:39:00.531436 11293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:58867
I20260620 18:39:00.536294 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:39:00 dist-test-slave-wp6g krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:39:00 dist-test-slave-wp6g krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.250.65@KRBTEST.COM
I20260620 18:39:00.561192 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.552436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.10.250.65:35749)
0620 18:39:00.552615 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:00.552622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:00.552635 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:39:00.552663 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:00.552665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:00.552678 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:00.552750 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:00.553335 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.554103 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.555045 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.555392 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.555460 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.557944 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:00.557955 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:00.557955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:00.557963 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:00.559822 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.560273 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.560278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.560278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.560359 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.560604 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.560606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.560606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.560814 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:00.560961 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:00.561025 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 18:39:00.561107 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
W20260620 18:39:00.561653 11398 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55560
Jun 20 18:39:00 dist-test-slave-wp6g krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:39:00 dist-test-slave-wp6g krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.65@KRBTEST.COM
I20260620 18:39:00.586220 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.577742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.10.250.65:35749)
0620 18:39:00.577848 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:00.577850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:00.578004 (+ 154us) server_negotiation.cc:407] Connection header received
0620 18:39:00.578037 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:00.578038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:00.578053 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:00.578116 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:00.578926 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.579435 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.580288 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.580509 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.580574 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.583087 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:00.583089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:00.583089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:00.583098 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:00.584653 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.585179 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.585181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.585181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.585290 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.585556 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.585557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.585557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.585675 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:00.585776 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:00.586025 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 18:39:00.586112 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:39:00.586905 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 11355
I20260620 18:39:00.591712 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 11489
I20260620 18:39:00.597325 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 11623
I20260620 18:39:00.601630 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 11261
2026-06-20T18:39:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1921 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:00 dist-test-slave-wp6g krb5kdc[11766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:00 dist-test-slave-wp6g krb5kdc[11766](info): set up 2 sockets
Jun 20 18:39:00 dist-test-slave-wp6g krb5kdc[11766](info): commencing operation
krb5kdc: starting...
W20260620 18:39:01.638200 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:39:01 dist-test-slave-wp6g krb5kdc[11766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:39:01.781654 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44437
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:40619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:44437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:01.865124 11779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:01.865342 11779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:01.865367 11779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:01.866667 11779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:01.866714 11779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:01.866729 11779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:01.866741 11779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:01.866753 11779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:01.868300 11779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:40619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:44437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:01.868587 11779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:01.868835 11779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:39:01.874190 11779 server_base.cc:1034] running on GCE node
W20260620 18:39:01.874172 11787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:01.874197 11784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:01.874164 11785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:01.874698 11779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:01.875064 11779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:01.876230 11779 hybrid_clock.cc:648] HybridClock initialized: now 1781980741876223 us; error 51 us; skew 500 ppm
Jun 20 18:39:01 dist-test-slave-wp6g krb5kdc[11766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980741, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:01.878826 11779 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:39:01.879312 11779 webserver.cc:460] Webserver started at http://127.10.250.126:41261/ using document root <none> and password file <none>
I20260620 18:39:01.879503 11779 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:01.879545 11779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:01.879649 11779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:01.880695 11779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "9f3482a47d2744f2848aa28dc1c087d8"
format_stamp: "Formatted at 2026-06-20 18:39:01 on dist-test-slave-wp6g"
server_key: "60f445be6b1e4b6e0366f62364190945"
server_key_iv: "e2c706979b1ad72b9d72d1284cc972a4"
server_key_version: "encryptionkey@0"
I20260620 18:39:01.881023 11779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "9f3482a47d2744f2848aa28dc1c087d8"
format_stamp: "Formatted at 2026-06-20 18:39:01 on dist-test-slave-wp6g"
server_key: "60f445be6b1e4b6e0366f62364190945"
server_key_iv: "e2c706979b1ad72b9d72d1284cc972a4"
server_key_version: "encryptionkey@0"
I20260620 18:39:01.882360 11779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:39:01.883179 11794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:01.883355 11779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:01.883459 11779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "9f3482a47d2744f2848aa28dc1c087d8"
format_stamp: "Formatted at 2026-06-20 18:39:01 on dist-test-slave-wp6g"
server_key: "60f445be6b1e4b6e0366f62364190945"
server_key_iv: "e2c706979b1ad72b9d72d1284cc972a4"
server_key_version: "encryptionkey@0"
I20260620 18:39:01.883541 11779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:01.913538 11779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:01.916306 11779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:01.916513 11779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:01.920883 11779 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:44437
I20260620 18:39:01.920915 11846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:44437 every 8 connection(s)
I20260620 18:39:01.921317 11779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:01.922266 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:01.924839 11847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8: Bootstrap starting.
I20260620 18:39:01.925364 11847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:01.925611 11847 log.cc:824] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:01.926290 11847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8: No bootstrap required, opened a new log
I20260620 18:39:01.927336 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 11779
I20260620 18:39:01.927439 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:01.927585 11241 external_mini_cluster.cc:1428] Setting key 4ade6f9441346144294cdc094e33236f
I20260620 18:39:01.928326 11847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3482a47d2744f2848aa28dc1c087d8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44437 } }
I20260620 18:39:01.928493 11847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:01.928537 11847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3482a47d2744f2848aa28dc1c087d8, State: Initialized, Role: FOLLOWER
I20260620 18:39:01.928721 11847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3482a47d2744f2848aa28dc1c087d8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44437 } }
I20260620 18:39:01.928822 11847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:01.928858 11847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:01.928913 11847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:01.929960 11847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3482a47d2744f2848aa28dc1c087d8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44437 } }
I20260620 18:39:01.930047 11847 leader_election.cc:302] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f3482a47d2744f2848aa28dc1c087d8; no voters:
I20260620 18:39:01.930260 11847 leader_election.cc:288] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:01.930281 11852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:01.930389 11852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [term 1 LEADER]: Becoming Leader. State: Replica: 9f3482a47d2744f2848aa28dc1c087d8, State: Running, Role: LEADER
I20260620 18:39:01.930473 11852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3482a47d2744f2848aa28dc1c087d8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44437 } }
I20260620 18:39:01.930554 11847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:01.930850 11853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f3482a47d2744f2848aa28dc1c087d8. Latest consensus state: current_term: 1 leader_uuid: "9f3482a47d2744f2848aa28dc1c087d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3482a47d2744f2848aa28dc1c087d8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44437 } } }
I20260620 18:39:01.930956 11853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:01.931137 11854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f3482a47d2744f2848aa28dc1c087d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3482a47d2744f2848aa28dc1c087d8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44437 } } }
I20260620 18:39:01.931213 11854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:01.931329 11858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:01.931856 11858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:01.933825 11858 catalog_manager.cc:1347] Generated new cluster ID: a679c496c77f4fe2a4f90e19b4ab8e98
I20260620 18:39:01.933883 11858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:39:01 dist-test-slave-wp6g krb5kdc[11766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:01.939949 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:01.928142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.10.250.126:44437)
0620 18:39:01.928327 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:01.928334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:01.928364 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:39:01.928469 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:01.928474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:01.928548 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:01.928695 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:01.929434 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:01.930599 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:01.931715 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:01.932413 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:01.932851 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:01.935420 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:01.935423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:01.935424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:01.935436 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:01.937894 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:01.938296 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:01.938299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:01.938299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:01.938412 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:01.938629 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:01.938630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:01.938630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:01.938949 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:01.939510 (+ 561us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:01.939582 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 18:39:01.939721 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260620 18:39:01.950133 11858 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:01.950763 11858 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:01.959956 11858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f3482a47d2744f2848aa28dc1c087d8: Generated new TSK 0
I20260620 18:39:01.960178 11858 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:39:01.989779 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44437
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:40619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:02.075096 11873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:02.075320 11873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:02.075345 11873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:02.076514 11873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:02.076553 11873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:02.076586 11873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:02.078145 11873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:40619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44437
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:02.078447 11873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:02.078763 11873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:02.084156 11878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:02.084288 11873 server_base.cc:1034] running on GCE node
W20260620 18:39:02.084196 11881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:02.084223 11879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:02.084661 11873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:02.084939 11873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:02.086086 11873 hybrid_clock.cc:648] HybridClock initialized: now 1781980742086067 us; error 34 us; skew 500 ppm
Jun 20 18:39:02 dist-test-slave-wp6g krb5kdc[11766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980742, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:02.088898 11873 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:39:02.089438 11873 webserver.cc:460] Webserver started at http://127.10.250.65:39309/ using document root <none> and password file <none>
I20260620 18:39:02.089676 11873 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:02.089747 11873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:02.089901 11873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:02.091069 11873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "c48d6892eed44ee6ac27eb1892ab7a32"
format_stamp: "Formatted at 2026-06-20 18:39:02 on dist-test-slave-wp6g"
server_key: "e08f190becff4b4207ec1d68bf1db362"
server_key_iv: "360a1205b7d1850c75297b86d9ef794a"
server_key_version: "encryptionkey@0"
I20260620 18:39:02.091444 11873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "c48d6892eed44ee6ac27eb1892ab7a32"
format_stamp: "Formatted at 2026-06-20 18:39:02 on dist-test-slave-wp6g"
server_key: "e08f190becff4b4207ec1d68bf1db362"
server_key_iv: "360a1205b7d1850c75297b86d9ef794a"
server_key_version: "encryptionkey@0"
I20260620 18:39:02.092784 11873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:39:02.093652 11888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:02.093845 11873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:02.093948 11873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "c48d6892eed44ee6ac27eb1892ab7a32"
format_stamp: "Formatted at 2026-06-20 18:39:02 on dist-test-slave-wp6g"
server_key: "e08f190becff4b4207ec1d68bf1db362"
server_key_iv: "360a1205b7d1850c75297b86d9ef794a"
server_key_version: "encryptionkey@0"
I20260620 18:39:02.094031 11873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:02.130582 11873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:02.133142 11873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:02.133324 11873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:02.133576 11873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:02.133937 11873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:02.133996 11873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:02.134038 11873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:02.134083 11873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:02.140194 11873 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:37453
I20260620 18:39:02.140249 12001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:37453 every 8 connection(s)
I20260620 18:39:02.140615 11873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:02.145318 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 11873
I20260620 18:39:02.145442 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:02.145605 11241 external_mini_cluster.cc:1428] Setting key caa53321c6d561682dc6374295379948
Jun 20 18:39:02 dist-test-slave-wp6g krb5kdc[11766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980742, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:02.151324 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.141479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:48631 (local address 127.10.250.126:44437)
0620 18:39:02.141529 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:02.141531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:02.141760 (+ 229us) server_negotiation.cc:407] Connection header received
0620 18:39:02.142816 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:02.142817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:02.142834 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:02.142885 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:02.143546 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.144065 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.145100 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.145332 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.145410 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.147811 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:02.147812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:02.147813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:02.147822 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:02.149708 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:02.150180 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:02.150182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:02.150182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:02.150259 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:02.150536 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:02.150538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:02.150538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:02.150699 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:02.150779 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:02.151049 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 18:39:02.151163 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:39:02.151355 12004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.141453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44437 (local address 127.10.250.65:48631)
0620 18:39:02.141632 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:02.141669 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:02.142699 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:02.142956 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:02.142961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:02.143248 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:02.143442 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:02.143446 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.144166 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.144167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:02.145004 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:02.145006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.145429 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.145430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:02.145579 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.146199 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:02.146213 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:02.147705 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:02.149822 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:02.149829 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:02.149831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:02.150094 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:02.150333 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:02.150333 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:02.150334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:02.150444 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:02.150773 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:02.150778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:02.150906 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 18:39:02.150996 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 18:39:02.151146 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260620 18:39:02.151950 12002 heartbeater.cc:347] Connected to a master server at 127.10.250.126:44437
I20260620 18:39:02.152065 12002 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:02.152284 12002 heartbeater.cc:511] Master 127.10.250.126:44437 requested a full tablet report, sending...
I20260620 18:39:02.152832 11811 ts_manager.cc:194] Registered new tserver with Master: c48d6892eed44ee6ac27eb1892ab7a32 (127.10.250.65:37453)
I20260620 18:39:02.153465 11811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:48631
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:39:02.175307 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44437
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:40619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:02.258160 12007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:02.258404 12007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:02.258428 12007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:02.259675 12007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:02.259727 12007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:02.259756 12007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:02.261317 12007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:40619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44437
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:02.261595 12007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:02.261837 12007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:02.267786 12013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:02.267742 12012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:02.267742 12015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:02.267776 12007 server_base.cc:1034] running on GCE node
I20260620 18:39:02.268198 12007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:02.268486 12007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:02.269654 12007 hybrid_clock.cc:648] HybridClock initialized: now 1781980742269634 us; error 44 us; skew 500 ppm
Jun 20 18:39:02 dist-test-slave-wp6g krb5kdc[11766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980742, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:02.272456 12007 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:39:02.272960 12007 webserver.cc:460] Webserver started at http://127.10.250.66:38215/ using document root <none> and password file <none>
I20260620 18:39:02.273267 12007 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:02.273342 12007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:02.273504 12007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:02.274667 12007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "5c3905c9548d48889dde45175d632d86"
format_stamp: "Formatted at 2026-06-20 18:39:02 on dist-test-slave-wp6g"
server_key: "550ebea97e7aca6f496de40fce4d930a"
server_key_iv: "edfbaab76fbba41b654249f9aecfb9d2"
server_key_version: "encryptionkey@0"
I20260620 18:39:02.275050 12007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "5c3905c9548d48889dde45175d632d86"
format_stamp: "Formatted at 2026-06-20 18:39:02 on dist-test-slave-wp6g"
server_key: "550ebea97e7aca6f496de40fce4d930a"
server_key_iv: "edfbaab76fbba41b654249f9aecfb9d2"
server_key_version: "encryptionkey@0"
I20260620 18:39:02.276464 12007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:39:02.277318 12022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:02.277540 12007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:02.277659 12007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "5c3905c9548d48889dde45175d632d86"
format_stamp: "Formatted at 2026-06-20 18:39:02 on dist-test-slave-wp6g"
server_key: "550ebea97e7aca6f496de40fce4d930a"
server_key_iv: "edfbaab76fbba41b654249f9aecfb9d2"
server_key_version: "encryptionkey@0"
I20260620 18:39:02.277746 12007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:02.305419 12007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:02.307798 12007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:02.307971 12007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:02.308216 12007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:02.308532 12007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:02.308595 12007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:02.308638 12007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:02.308665 12007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:02.314771 12007 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:35967
I20260620 18:39:02.314826 12135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:35967 every 8 connection(s)
I20260620 18:39:02.315171 12007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:02.320672 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 12007
I20260620 18:39:02.320755 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:02.320956 11241 external_mini_cluster.cc:1428] Setting key 7f2494835450e0456347ce25e467b920
Jun 20 18:39:02 dist-test-slave-wp6g krb5kdc[11766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980742, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:02.327210 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.315977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44437 (local address 127.10.250.66:34913)
0620 18:39:02.316232 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:02.316282 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:02.317371 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:02.317684 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:02.317689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:02.317993 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:02.318213 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:02.318217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.319111 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.319112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:02.320019 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:02.320020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.320494 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.320495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:02.320631 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.321334 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:02.321344 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:02.323684 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:02.325705 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:02.325713 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:02.325715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:02.325996 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:02.326231 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:02.326231 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:02.326232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:02.326337 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:02.326683 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:02.326688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:02.326811 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 18:39:02.326898 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 18:39:02.327024 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260620 18:39:02.327206 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.316029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:34913 (local address 127.10.250.126:44437)
0620 18:39:02.316126 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:02.316129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:02.316391 (+ 262us) server_negotiation.cc:407] Connection header received
0620 18:39:02.317515 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:02.317517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:02.317534 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:02.317600 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:02.318326 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.319003 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.320127 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.320362 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.320462 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.323794 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:02.323795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:02.323796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:02.323803 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:02.325579 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:02.326083 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:02.326085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:02.326085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:02.326158 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:02.326407 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:02.326408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:02.326409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:02.326599 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:02.326683 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:02.326951 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 18:39:02.327062 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:39:02.327735 12136 heartbeater.cc:347] Connected to a master server at 127.10.250.126:44437
I20260620 18:39:02.327854 12136 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:02.328094 12136 heartbeater.cc:511] Master 127.10.250.126:44437 requested a full tablet report, sending...
I20260620 18:39:02.328507 11811 ts_manager.cc:194] Registered new tserver with Master: 5c3905c9548d48889dde45175d632d86 (127.10.250.66:35967)
I20260620 18:39:02.328908 11811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:34913
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:39:02.350335 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44437
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:40619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:02.434269 12141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:02.434554 12141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:02.434602 12141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:02.435851 12141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:02.435931 12141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:02.435981 12141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:02.437553 12141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:40619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44437
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:02.437841 12141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:02.438081 12141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:02.443145 12149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:02.443204 12141 server_base.cc:1034] running on GCE node
W20260620 18:39:02.443099 12146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:02.443118 12147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:02.443533 12141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:02.443804 12141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:02.444984 12141 hybrid_clock.cc:648] HybridClock initialized: now 1781980742444976 us; error 34 us; skew 500 ppm
Jun 20 18:39:02 dist-test-slave-wp6g krb5kdc[11766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980742, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:02.447789 12141 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:39:02.448315 12141 webserver.cc:460] Webserver started at http://127.10.250.67:37365/ using document root <none> and password file <none>
I20260620 18:39:02.448547 12141 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:02.448618 12141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:02.448776 12141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:02.449894 12141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "8974323e98f944b1ba027b1a5beb4a79"
format_stamp: "Formatted at 2026-06-20 18:39:02 on dist-test-slave-wp6g"
server_key: "cbc1bd97f8a2ce5d31bd93f03e642fe7"
server_key_iv: "f728ff58fde1081e8c5bd9f6cf295352"
server_key_version: "encryptionkey@0"
I20260620 18:39:02.450371 12141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "8974323e98f944b1ba027b1a5beb4a79"
format_stamp: "Formatted at 2026-06-20 18:39:02 on dist-test-slave-wp6g"
server_key: "cbc1bd97f8a2ce5d31bd93f03e642fe7"
server_key_iv: "f728ff58fde1081e8c5bd9f6cf295352"
server_key_version: "encryptionkey@0"
I20260620 18:39:02.451901 12141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:02.452816 12156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:02.453068 12141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:02.453179 12141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "8974323e98f944b1ba027b1a5beb4a79"
format_stamp: "Formatted at 2026-06-20 18:39:02 on dist-test-slave-wp6g"
server_key: "cbc1bd97f8a2ce5d31bd93f03e642fe7"
server_key_iv: "f728ff58fde1081e8c5bd9f6cf295352"
server_key_version: "encryptionkey@0"
I20260620 18:39:02.453259 12141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:02.471933 12141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:02.473944 12141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:02.474124 12141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:02.474382 12141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:02.474797 12141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:02.474854 12141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:02.474890 12141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:02.474918 12141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:02.481079 12141 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:33141
I20260620 18:39:02.481117 12269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:33141 every 8 connection(s)
I20260620 18:39:02.481479 12141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:02.485664 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 12141
I20260620 18:39:02.485777 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:02.485957 11241 external_mini_cluster.cc:1428] Setting key e1eb97bdd288e4771b97b9da144e05cd
Jun 20 18:39:02 dist-test-slave-wp6g krb5kdc[11766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980742, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:02.491544 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.482308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:43695 (local address 127.10.250.126:44437)
0620 18:39:02.482382 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:02.482384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:02.482596 (+ 212us) server_negotiation.cc:407] Connection header received
0620 18:39:02.483561 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:02.483562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:02.483577 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:02.483626 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:02.484311 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.484826 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.485988 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.486302 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.486381 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.488794 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:02.488795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:02.488795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:02.488803 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:02.490242 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:02.490706 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:02.490707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:02.490707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:02.490762 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:02.490982 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:02.490983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:02.490983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:02.491124 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:02.491174 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:02.491415 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 18:39:02.491476 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 18:39:02.491760 12272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.482246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44437 (local address 127.10.250.67:43695)
0620 18:39:02.482408 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:02.482453 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:02.483463 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:02.483692 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:02.483696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:02.483961 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:02.484204 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:02.484210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.484938 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.484939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:02.485897 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:02.485901 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.486399 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.486400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:02.486617 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.487229 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:02.487238 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:02.488708 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:02.490324 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:02.490330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:02.490332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:02.490642 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:02.490822 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:02.490823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:02.490823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:02.490923 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:02.491178 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:02.491182 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:02.491297 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 18:39:02.491374 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 18:39:02.491495 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260620 18:39:02.492380 12270 heartbeater.cc:347] Connected to a master server at 127.10.250.126:44437
I20260620 18:39:02.492497 12270 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:02.492695 12270 heartbeater.cc:511] Master 127.10.250.126:44437 requested a full tablet report, sending...
I20260620 18:39:02.493083 11811 ts_manager.cc:194] Registered new tserver with Master: 8974323e98f944b1ba027b1a5beb4a79 (127.10.250.67:33141)
I20260620 18:39:02.493598 11811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:43695
I20260620 18:39:02.497826 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:39:02.506402 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.499553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.10.250.126:44437)
0620 18:39:02.499687 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:02.499690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:02.499808 (+ 118us) server_negotiation.cc:407] Connection header received
0620 18:39:02.499897 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:02.499898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:02.499908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:02.499966 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:02.500575 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.501078 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.501829 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.502028 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.502115 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.503602 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:02.503603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:02.503604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:02.503612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:02.505034 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:02.505480 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:02.505482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:02.505482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:02.505550 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:02.505779 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:02.505780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:02.505781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:02.505951 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:02.506068 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:02.506205 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 18:39:02.506294 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:39:02.508812 11811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:02.510488 11811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:02.517782 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.513430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48634 (local address 127.10.250.65:37453)
0620 18:39:02.513637 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:02.513642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:02.513655 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:39:02.513709 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:02.513711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:02.513730 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:02.513877 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:02.514087 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.515321 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.516807 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.517416 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.517488 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.517574 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:02.517658 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 18:39:02.517698 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":87,"threads_started":1}
I20260620 18:39:02.517797 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.513295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:37453 (local address 127.0.0.1:48634)
0620 18:39:02.513462 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:02.513502 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:02.513567 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:02.513885 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:02.513889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:02.513892 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:02.513980 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:02.513982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.515432 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.515433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:02.516715 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:02.516717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.517492 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.517493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:02.517563 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.517567 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:39:02.517643 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 18:39:02.517691 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":48,"thread_start_us":61,"threads_started":1}
I20260620 18:39:02.517874 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.513719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45970 (local address 127.10.250.66:35967)
0620 18:39:02.513881 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:02.513886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:02.514070 (+ 184us) server_negotiation.cc:407] Connection header received
0620 18:39:02.514152 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:02.514155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:02.514171 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:02.514241 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:02.514403 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.515498 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.516513 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.517264 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.517351 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.517409 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:02.517742 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 18:39:02.517793 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260620 18:39:02.517701 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.513494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:35967 (local address 127.0.0.1:45970)
0620 18:39:02.513998 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:02.514021 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:02.514079 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:02.514254 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:02.514255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:02.514257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:02.514300 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:02.514302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.515624 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.515625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:02.516397 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:02.516399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.517368 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.517369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:02.517461 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.517480 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 18:39:02.517552 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:39:02.517616 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":76,"threads_started":1}
I20260620 18:39:02.518097 12284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.513295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:33141 (local address 127.0.0.1:58232)
0620 18:39:02.513904 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:02.513913 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:02.513971 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:02.514241 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:02.514242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:02.514243 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:02.514300 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:02.514303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.515361 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.515362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:02.516868 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:02.516870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.517845 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.517846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:02.517910 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.517914 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:39:02.517971 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 18:39:02.518012 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":42,"thread_start_us":75,"threads_started":1}
I20260620 18:39:02.518350 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.513521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.10.250.67:33141)
0620 18:39:02.513719 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:02.513723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:02.513973 (+ 250us) server_negotiation.cc:407] Connection header received
0620 18:39:02.514044 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:02.514046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:02.514072 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:02.514145 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:02.514360 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.515256 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.516952 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.517687 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.517756 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.517806 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:02.518231 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 18:39:02.518278 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":93,"threads_started":1}
I20260620 18:39:02.519083 12070 tablet_service.cc:1467] Processing CreateTablet for tablet e6954f9c8ed64cb6990c0876d1eec231 (DEFAULT_TABLE table=test-table [id=30900329f6b64d6a836e6df7303be5d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:02.519112 12204 tablet_service.cc:1467] Processing CreateTablet for tablet e6954f9c8ed64cb6990c0876d1eec231 (DEFAULT_TABLE table=test-table [id=30900329f6b64d6a836e6df7303be5d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:02.519112 11936 tablet_service.cc:1467] Processing CreateTablet for tablet e6954f9c8ed64cb6990c0876d1eec231 (DEFAULT_TABLE table=test-table [id=30900329f6b64d6a836e6df7303be5d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:02.519438 12070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6954f9c8ed64cb6990c0876d1eec231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:02.519440 11936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6954f9c8ed64cb6990c0876d1eec231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:02.519454 12204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6954f9c8ed64cb6990c0876d1eec231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:02.522074 12287 tablet_bootstrap.cc:492] T e6954f9c8ed64cb6990c0876d1eec231 P 5c3905c9548d48889dde45175d632d86: Bootstrap starting.
I20260620 18:39:02.522837 12287 tablet_bootstrap.cc:654] T e6954f9c8ed64cb6990c0876d1eec231 P 5c3905c9548d48889dde45175d632d86: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:02.522825 12288 tablet_bootstrap.cc:492] T e6954f9c8ed64cb6990c0876d1eec231 P 8974323e98f944b1ba027b1a5beb4a79: Bootstrap starting.
I20260620 18:39:02.523159 12289 tablet_bootstrap.cc:492] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32: Bootstrap starting.
I20260620 18:39:02.523490 12287 log.cc:824] T e6954f9c8ed64cb6990c0876d1eec231 P 5c3905c9548d48889dde45175d632d86: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:02.523499 12288 tablet_bootstrap.cc:654] T e6954f9c8ed64cb6990c0876d1eec231 P 8974323e98f944b1ba027b1a5beb4a79: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:02.523862 12289 tablet_bootstrap.cc:654] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:02.524174 12288 log.cc:824] T e6954f9c8ed64cb6990c0876d1eec231 P 8974323e98f944b1ba027b1a5beb4a79: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:02.524328 12287 tablet_bootstrap.cc:492] T e6954f9c8ed64cb6990c0876d1eec231 P 5c3905c9548d48889dde45175d632d86: No bootstrap required, opened a new log
I20260620 18:39:02.524406 12287 ts_tablet_manager.cc:1397] T e6954f9c8ed64cb6990c0876d1eec231 P 5c3905c9548d48889dde45175d632d86: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:39:02.524430 12289 log.cc:824] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:02.524937 12288 tablet_bootstrap.cc:492] T e6954f9c8ed64cb6990c0876d1eec231 P 8974323e98f944b1ba027b1a5beb4a79: No bootstrap required, opened a new log
I20260620 18:39:02.525015 12288 ts_tablet_manager.cc:1397] T e6954f9c8ed64cb6990c0876d1eec231 P 8974323e98f944b1ba027b1a5beb4a79: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:39:02.525460 12289 tablet_bootstrap.cc:492] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32: No bootstrap required, opened a new log
I20260620 18:39:02.525529 12289 ts_tablet_manager.cc:1397] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:39:02.526285 12287 raft_consensus.cc:348] T e6954f9c8ed64cb6990c0876d1eec231 P 5c3905c9548d48889dde45175d632d86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d6892eed44ee6ac27eb1892ab7a32" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37453 } } peers { permanent_uuid: "8974323e98f944b1ba027b1a5beb4a79" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33141 } } peers { permanent_uuid: "5c3905c9548d48889dde45175d632d86" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35967 } }
I20260620 18:39:02.526463 12287 raft_consensus.cc:374] T e6954f9c8ed64cb6990c0876d1eec231 P 5c3905c9548d48889dde45175d632d86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:02.526510 12287 raft_consensus.cc:729] T e6954f9c8ed64cb6990c0876d1eec231 P 5c3905c9548d48889dde45175d632d86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c3905c9548d48889dde45175d632d86, State: Initialized, Role: FOLLOWER
I20260620 18:39:02.526688 12287 consensus_queue.cc:260] T e6954f9c8ed64cb6990c0876d1eec231 P 5c3905c9548d48889dde45175d632d86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d6892eed44ee6ac27eb1892ab7a32" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37453 } } peers { permanent_uuid: "8974323e98f944b1ba027b1a5beb4a79" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33141 } } peers { permanent_uuid: "5c3905c9548d48889dde45175d632d86" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35967 } }
I20260620 18:39:02.527022 12136 heartbeater.cc:503] Master 127.10.250.126:44437 was elected leader, sending a full tablet report...
I20260620 18:39:02.527029 12287 ts_tablet_manager.cc:1428] T e6954f9c8ed64cb6990c0876d1eec231 P 5c3905c9548d48889dde45175d632d86: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:02.527078 12288 raft_consensus.cc:348] T e6954f9c8ed64cb6990c0876d1eec231 P 8974323e98f944b1ba027b1a5beb4a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d6892eed44ee6ac27eb1892ab7a32" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37453 } } peers { permanent_uuid: "8974323e98f944b1ba027b1a5beb4a79" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33141 } } peers { permanent_uuid: "5c3905c9548d48889dde45175d632d86" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35967 } }
I20260620 18:39:02.527240 12288 raft_consensus.cc:374] T e6954f9c8ed64cb6990c0876d1eec231 P 8974323e98f944b1ba027b1a5beb4a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:02.527294 12288 raft_consensus.cc:729] T e6954f9c8ed64cb6990c0876d1eec231 P 8974323e98f944b1ba027b1a5beb4a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8974323e98f944b1ba027b1a5beb4a79, State: Initialized, Role: FOLLOWER
I20260620 18:39:02.527417 12288 consensus_queue.cc:260] T e6954f9c8ed64cb6990c0876d1eec231 P 8974323e98f944b1ba027b1a5beb4a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d6892eed44ee6ac27eb1892ab7a32" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37453 } } peers { permanent_uuid: "8974323e98f944b1ba027b1a5beb4a79" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33141 } } peers { permanent_uuid: "5c3905c9548d48889dde45175d632d86" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35967 } }
I20260620 18:39:02.527571 12289 raft_consensus.cc:348] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d6892eed44ee6ac27eb1892ab7a32" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37453 } } peers { permanent_uuid: "8974323e98f944b1ba027b1a5beb4a79" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33141 } } peers { permanent_uuid: "5c3905c9548d48889dde45175d632d86" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35967 } }
I20260620 18:39:02.527717 12288 ts_tablet_manager.cc:1428] T e6954f9c8ed64cb6990c0876d1eec231 P 8974323e98f944b1ba027b1a5beb4a79: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:02.527740 12289 raft_consensus.cc:374] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:02.527734 12270 heartbeater.cc:503] Master 127.10.250.126:44437 was elected leader, sending a full tablet report...
I20260620 18:39:02.527776 12289 raft_consensus.cc:729] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c48d6892eed44ee6ac27eb1892ab7a32, State: Initialized, Role: FOLLOWER
I20260620 18:39:02.527886 12289 consensus_queue.cc:260] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d6892eed44ee6ac27eb1892ab7a32" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37453 } } peers { permanent_uuid: "8974323e98f944b1ba027b1a5beb4a79" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33141 } } peers { permanent_uuid: "5c3905c9548d48889dde45175d632d86" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35967 } }
I20260620 18:39:02.528175 12289 ts_tablet_manager.cc:1428] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:02.528182 12002 heartbeater.cc:503] Master 127.10.250.126:44437 was elected leader, sending a full tablet report...
W20260620 18:39:02.565833 12137 tablet.cc:2369] T e6954f9c8ed64cb6990c0876d1eec231 P 5c3905c9548d48889dde45175d632d86: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:39:02.641409 12003 tablet.cc:2369] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:39:02.732117 12271 tablet.cc:2369] T e6954f9c8ed64cb6990c0876d1eec231 P 8974323e98f944b1ba027b1a5beb4a79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:39:02.831925 12295 raft_consensus.cc:482] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:02.832134 12295 raft_consensus.cc:504] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d6892eed44ee6ac27eb1892ab7a32" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37453 } } peers { permanent_uuid: "8974323e98f944b1ba027b1a5beb4a79" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33141 } } peers { permanent_uuid: "5c3905c9548d48889dde45175d632d86" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35967 } }
I20260620 18:39:02.832512 12295 leader_election.cc:288] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8974323e98f944b1ba027b1a5beb4a79 (127.10.250.67:33141), 5c3905c9548d48889dde45175d632d86 (127.10.250.66:35967)
I20260620 18:39:02.835728 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.833118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:46729 (local address 127.10.250.66:35967)
0620 18:39:02.833159 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:02.833161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:02.833169 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:39:02.833196 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:02.833198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:02.833229 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:02.833288 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:02.833488 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.833951 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.835026 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.835373 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.835428 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.835457 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:02.835608 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 18:39:02.835654 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:39:02.835762 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.832895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:41299 (local address 127.10.250.67:33141)
0620 18:39:02.832974 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:02.832976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:02.833053 (+ 77us) server_negotiation.cc:407] Connection header received
0620 18:39:02.833167 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:02.833168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:02.833178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:02.833230 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:02.833447 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.833945 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.834987 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.835383 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.835473 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.835481 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:02.835656 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 18:39:02.835694 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 18:39:02.835772 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.832984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:35967 (local address 127.10.250.65:46729)
0620 18:39:02.833061 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:02.833071 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:02.833126 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:02.833307 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:02.833308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:02.833310 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:02.833390 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:02.833391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.834055 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.834056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:02.834911 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:02.834914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.835457 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.835458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:02.835535 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.835539 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:39:02.835587 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 18:39:02.835642 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":2,"thread_start_us":38,"threads_started":1}
I20260620 18:39:02.835789 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.832784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:33141 (local address 127.10.250.65:41299)
0620 18:39:02.832953 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:02.832969 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:02.833090 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:02.833263 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:02.833264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:02.833266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:02.833375 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:02.833377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.834034 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.834035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:02.834902 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:02.834905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.835486 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.835487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:02.835560 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.835582 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:39:02.835640 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 18:39:02.835678 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":22,"thread_start_us":77,"threads_started":1}
I20260620 18:39:02.836190 12090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6954f9c8ed64cb6990c0876d1eec231" candidate_uuid: "c48d6892eed44ee6ac27eb1892ab7a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c3905c9548d48889dde45175d632d86" is_pre_election: true
I20260620 18:39:02.836287 12224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6954f9c8ed64cb6990c0876d1eec231" candidate_uuid: "c48d6892eed44ee6ac27eb1892ab7a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974323e98f944b1ba027b1a5beb4a79" is_pre_election: true
I20260620 18:39:02.836369 12090 raft_consensus.cc:2454] T e6954f9c8ed64cb6990c0876d1eec231 P 5c3905c9548d48889dde45175d632d86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c48d6892eed44ee6ac27eb1892ab7a32 in term 0.
I20260620 18:39:02.836406 12224 raft_consensus.cc:2454] T e6954f9c8ed64cb6990c0876d1eec231 P 8974323e98f944b1ba027b1a5beb4a79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c48d6892eed44ee6ac27eb1892ab7a32 in term 0.
I20260620 18:39:02.836606 11889 leader_election.cc:302] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c3905c9548d48889dde45175d632d86, c48d6892eed44ee6ac27eb1892ab7a32; no voters:
I20260620 18:39:02.836735 12295 raft_consensus.cc:2781] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:02.836831 12295 raft_consensus.cc:482] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:02.836865 12295 raft_consensus.cc:3037] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:02.837581 12295 raft_consensus.cc:504] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d6892eed44ee6ac27eb1892ab7a32" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37453 } } peers { permanent_uuid: "8974323e98f944b1ba027b1a5beb4a79" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33141 } } peers { permanent_uuid: "5c3905c9548d48889dde45175d632d86" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35967 } }
I20260620 18:39:02.837734 12295 leader_election.cc:288] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [CANDIDATE]: Term 1 election: Requested vote from peers 8974323e98f944b1ba027b1a5beb4a79 (127.10.250.67:33141), 5c3905c9548d48889dde45175d632d86 (127.10.250.66:35967)
I20260620 18:39:02.837970 12224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6954f9c8ed64cb6990c0876d1eec231" candidate_uuid: "c48d6892eed44ee6ac27eb1892ab7a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8974323e98f944b1ba027b1a5beb4a79"
I20260620 18:39:02.838006 12090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6954f9c8ed64cb6990c0876d1eec231" candidate_uuid: "c48d6892eed44ee6ac27eb1892ab7a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c3905c9548d48889dde45175d632d86"
I20260620 18:39:02.838079 12224 raft_consensus.cc:3037] T e6954f9c8ed64cb6990c0876d1eec231 P 8974323e98f944b1ba027b1a5beb4a79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:02.838124 12090 raft_consensus.cc:3037] T e6954f9c8ed64cb6990c0876d1eec231 P 5c3905c9548d48889dde45175d632d86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:02.839017 12224 raft_consensus.cc:2454] T e6954f9c8ed64cb6990c0876d1eec231 P 8974323e98f944b1ba027b1a5beb4a79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c48d6892eed44ee6ac27eb1892ab7a32 in term 1.
I20260620 18:39:02.839027 12090 raft_consensus.cc:2454] T e6954f9c8ed64cb6990c0876d1eec231 P 5c3905c9548d48889dde45175d632d86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c48d6892eed44ee6ac27eb1892ab7a32 in term 1.
I20260620 18:39:02.839274 11889 leader_election.cc:302] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c3905c9548d48889dde45175d632d86, c48d6892eed44ee6ac27eb1892ab7a32; no voters:
I20260620 18:39:02.839428 12295 raft_consensus.cc:2781] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:02.839617 12295 raft_consensus.cc:686] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [term 1 LEADER]: Becoming Leader. State: Replica: c48d6892eed44ee6ac27eb1892ab7a32, State: Running, Role: LEADER
I20260620 18:39:02.839741 12295 consensus_queue.cc:237] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d6892eed44ee6ac27eb1892ab7a32" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37453 } } peers { permanent_uuid: "8974323e98f944b1ba027b1a5beb4a79" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33141 } } peers { permanent_uuid: "5c3905c9548d48889dde45175d632d86" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35967 } }
I20260620 18:39:02.840607 11810 catalog_manager.cc:5495] T e6954f9c8ed64cb6990c0876d1eec231 P c48d6892eed44ee6ac27eb1892ab7a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to c48d6892eed44ee6ac27eb1892ab7a32 (127.10.250.65). New cstate: current_term: 1 leader_uuid: "c48d6892eed44ee6ac27eb1892ab7a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d6892eed44ee6ac27eb1892ab7a32" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8974323e98f944b1ba027b1a5beb4a79" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c3905c9548d48889dde45175d632d86" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35967 } health_report { overall_health: UNKNOWN } } }
Jun 20 18:39:02 dist-test-slave-wp6g krb5kdc[11766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.65@KRBTEST.COM
I20260620 18:39:02.880329 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.872407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.10.250.65:37453)
0620 18:39:02.872504 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:02.872507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:02.872637 (+ 130us) server_negotiation.cc:407] Connection header received
0620 18:39:02.872727 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:02.872728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:02.872740 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:02.872818 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:02.873412 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.873917 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.874738 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.874931 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.875037 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.877532 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:02.877547 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:02.877547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:02.877555 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:02.879283 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:02.879621 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:02.879625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:02.879625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:02.879686 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:02.879854 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:02.879854 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:02.879855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:02.879996 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:02.880120 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:02.880197 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 18:39:02.880260 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 18:39:02 dist-test-slave-wp6g krb5kdc[11766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980742, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:39:02 dist-test-slave-wp6g krb5kdc[11766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980742, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.250.65@KRBTEST.COM
I20260620 18:39:02.905107 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.897291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48642 (local address 127.10.250.65:37453)
0620 18:39:02.897357 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:02.897360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:02.897443 (+ 83us) server_negotiation.cc:407] Connection header received
0620 18:39:02.897544 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:02.897545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:02.897556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:02.897652 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:02.898321 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.898888 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.899676 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.899878 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.899961 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.902298 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:02.902299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:02.902299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:02.902307 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:02.903991 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:02.904367 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:02.904368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:02.904369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:02.904434 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:02.904618 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:02.904618 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:02.904618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:02.904715 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:02.904796 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:02.904957 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 18:39:02.905031 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 18:39:02 dist-test-slave-wp6g krb5kdc[11766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:39:02 dist-test-slave-wp6g krb5kdc[11766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.65@KRBTEST.COM
I20260620 18:39:02.929023 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.921338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48658 (local address 127.10.250.65:37453)
0620 18:39:02.921410 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:02.921413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:02.921489 (+ 76us) server_negotiation.cc:407] Connection header received
0620 18:39:02.921578 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:02.921579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:02.921592 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:02.921664 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:02.922313 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.922891 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.923662 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:02.923854 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:02.923930 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:02.926346 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:02.926348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:02.926348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:02.926359 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:02.927825 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:02.928217 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:02.928219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:02.928219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:02.928279 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:02.928483 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:02.928484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:02.928484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:02.928603 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:02.928727 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:02.928824 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 18:39:02.928912 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 18:39:02.929538 11916 tablet_service.cc:2908] Rejecting scan request for tablet e6954f9c8ed64cb6990c0876d1eec231: Uninitialized: clean time has not yet been initialized
I20260620 18:39:02.930469 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 11873
I20260620 18:39:02.936614 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 12007
I20260620 18:39:02.941526 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 12141
I20260620 18:39:02.946100 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 11779
2026-06-20T18:39:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2345 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:03 dist-test-slave-wp6g krb5kdc[12312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:03 dist-test-slave-wp6g krb5kdc[12312](info): set up 2 sockets
Jun 20 18:39:03 dist-test-slave-wp6g krb5kdc[12312](info): commencing operation
krb5kdc: starting...
W20260620 18:39:04.092139 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 18:39:04.306629 11241 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.214s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 18:39:04.623595 11241 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.317s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 18:39:04.928970 11241 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.305s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/test-user.keytab.
W20260620 18:39:05.499949 11241 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.571s user 0.000s sys 0.001s
Jun 20 18:39:05 dist-test-slave-wp6g krb5kdc[12312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
W20260620 18:39:06.861369 11241 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.10.250.126: real 1.338s user 0.000s sys 0.003s
I20260620 18:39:06.861774 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:46539
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:43173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:46539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:06.949577 12325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:06.949819 12325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:06.949846 12325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:06.951148 12325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:06.951195 12325 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:39:06.951210 12325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:06.951222 12325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:06.951237 12325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:06.951249 12325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:06.952806 12325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:43173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:46539
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:46539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:06.953023 12325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:06.953197 12325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:06.958323 12333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:06.958328 12331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:06.958323 12330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:06.958606 12325 server_base.cc:1034] running on GCE node
I20260620 18:39:06.958881 12325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:06.959141 12325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:06.960284 12325 hybrid_clock.cc:648] HybridClock initialized: now 1781980746960265 us; error 41 us; skew 500 ppm
Jun 20 18:39:06 dist-test-slave-wp6g krb5kdc[12312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980746, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:06.968324 12325 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:39:06.968820 12325 webserver.cc:460] Webserver started at http://127.10.250.126:43297/ using document root <none> and password file <none>
I20260620 18:39:06.969018 12325 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:06.969061 12325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:06.969157 12325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:06.970234 12325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "46c34b012a0449d0bf42244ee9cacc5f"
format_stamp: "Formatted at 2026-06-20 18:39:06 on dist-test-slave-wp6g"
server_key: "5054ba76adbc9760c021c17cd8e5ae2b"
server_key_iv: "130f89a384648005f2f8d63790d76ddc"
server_key_version: "encryptionkey@0"
I20260620 18:39:06.970624 12325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "46c34b012a0449d0bf42244ee9cacc5f"
format_stamp: "Formatted at 2026-06-20 18:39:06 on dist-test-slave-wp6g"
server_key: "5054ba76adbc9760c021c17cd8e5ae2b"
server_key_iv: "130f89a384648005f2f8d63790d76ddc"
server_key_version: "encryptionkey@0"
I20260620 18:39:06.972038 12325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:06.972869 12340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:06.973187 12325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:06.973320 12325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "46c34b012a0449d0bf42244ee9cacc5f"
format_stamp: "Formatted at 2026-06-20 18:39:06 on dist-test-slave-wp6g"
server_key: "5054ba76adbc9760c021c17cd8e5ae2b"
server_key_iv: "130f89a384648005f2f8d63790d76ddc"
server_key_version: "encryptionkey@0"
I20260620 18:39:06.973423 12325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:06.987340 12325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:06.990196 12325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:06.990399 12325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:06.995591 12325 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:46539
I20260620 18:39:06.995659 12402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:46539 every 8 connection(s)
I20260620 18:39:06.996016 12325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:06.996739 12403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:06.997283 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 12325
I20260620 18:39:06.997401 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:06.997573 11241 external_mini_cluster.cc:1428] Setting key 7a7e905c8796bd4aea0beb56f2cf8401
I20260620 18:39:07.000218 12403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f: Bootstrap starting.
I20260620 18:39:07.000893 12403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:07.001226 12403 log.cc:824] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:07.001987 12403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f: No bootstrap required, opened a new log
I20260620 18:39:07.004192 12403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c34b012a0449d0bf42244ee9cacc5f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46539 } }
I20260620 18:39:07.004379 12403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:07.004433 12403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c34b012a0449d0bf42244ee9cacc5f, State: Initialized, Role: FOLLOWER
I20260620 18:39:07.004586 12403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c34b012a0449d0bf42244ee9cacc5f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46539 } }
I20260620 18:39:07.004673 12403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:07.004719 12403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:07.004769 12403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [term 0 FOLLOWER]: Advancing to term 1
Jun 20 18:39:07 dist-test-slave-wp6g krb5kdc[12312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:07.008991 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:06.998157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.10.250.126:46539)
0620 18:39:06.998332 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:06.998342 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:06.998598 (+ 256us) server_negotiation.cc:407] Connection header received
0620 18:39:06.999095 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:06.999102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:06.999162 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:06.999291 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:07.000159 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:07.001266 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:07.002126 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:07.002652 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:07.002788 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:07.005454 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:07.005460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:07.005460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:07.005474 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:07.007323 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:07.007746 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:07.007750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:07.007750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:07.007856 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:07.008074 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:07.008075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:07.008076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:07.008357 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:07.008518 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:07.008632 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 18:39:07.008798 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260620 18:39:07.033335 12403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c34b012a0449d0bf42244ee9cacc5f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46539 } }
I20260620 18:39:07.033684 12403 leader_election.cc:302] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46c34b012a0449d0bf42244ee9cacc5f; no voters:
I20260620 18:39:07.033936 12403 leader_election.cc:288] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:07.034119 12408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:07.034251 12403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:07.034312 12408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [term 1 LEADER]: Becoming Leader. State: Replica: 46c34b012a0449d0bf42244ee9cacc5f, State: Running, Role: LEADER
I20260620 18:39:07.034400 12408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c34b012a0449d0bf42244ee9cacc5f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46539 } }
I20260620 18:39:07.035097 12409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46c34b012a0449d0bf42244ee9cacc5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c34b012a0449d0bf42244ee9cacc5f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46539 } } }
I20260620 18:39:07.035210 12409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:07.035396 12410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46c34b012a0449d0bf42244ee9cacc5f. Latest consensus state: current_term: 1 leader_uuid: "46c34b012a0449d0bf42244ee9cacc5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c34b012a0449d0bf42244ee9cacc5f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46539 } } }
I20260620 18:39:07.035462 12410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:07.035698 12417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:07.036506 12417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:07.038379 12417 catalog_manager.cc:1347] Generated new cluster ID: 09700a88d3e44bf2b3e2a9843ddeb139
I20260620 18:39:07.038432 12417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:07.047900 12417 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:07.048552 12417 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:07.056113 12417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f: Generated new TSK 0
I20260620 18:39:07.056367 12417 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
W20260620 18:39:08.027364 11241 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.10.250.65: real 0.959s user 0.000s sys 0.003s
I20260620 18:39:08.027803 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:46539
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:43173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:08.113906 12429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:08.114169 12429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:08.114197 12429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:08.115475 12429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:08.115533 12429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:08.115558 12429 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:39:08.115578 12429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:08.117041 12429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:43173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.250.126:46539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:08.117259 12429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:08.117458 12429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:08.122844 12437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:08.122829 12429 server_base.cc:1034] running on GCE node
W20260620 18:39:08.122833 12435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:08.122869 12434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:08.123307 12429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:08.123543 12429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:08.124696 12429 hybrid_clock.cc:648] HybridClock initialized: now 1781980748124683 us; error 30 us; skew 500 ppm
Jun 20 18:39:08 dist-test-slave-wp6g krb5kdc[12312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:08.127398 12429 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:39:08.127868 12429 webserver.cc:460] Webserver started at http://127.10.250.65:40585/ using document root <none> and password file <none>
I20260620 18:39:08.128123 12429 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:08.128199 12429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:08.128373 12429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:08.129489 12429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "a3b3753324af4b47a13b9e2f99d4d669"
format_stamp: "Formatted at 2026-06-20 18:39:08 on dist-test-slave-wp6g"
server_key: "8558df50d1c65a7aade32d3ec5d9a5af"
server_key_iv: "616c20729aa068c33993b16d300ae62f"
server_key_version: "encryptionkey@0"
I20260620 18:39:08.129884 12429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "a3b3753324af4b47a13b9e2f99d4d669"
format_stamp: "Formatted at 2026-06-20 18:39:08 on dist-test-slave-wp6g"
server_key: "8558df50d1c65a7aade32d3ec5d9a5af"
server_key_iv: "616c20729aa068c33993b16d300ae62f"
server_key_version: "encryptionkey@0"
I20260620 18:39:08.131474 12429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:39:08.132298 12444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:08.132479 12429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:08.132591 12429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "a3b3753324af4b47a13b9e2f99d4d669"
format_stamp: "Formatted at 2026-06-20 18:39:08 on dist-test-slave-wp6g"
server_key: "8558df50d1c65a7aade32d3ec5d9a5af"
server_key_iv: "616c20729aa068c33993b16d300ae62f"
server_key_version: "encryptionkey@0"
I20260620 18:39:08.132674 12429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:08.153287 12429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:08.155467 12429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:08.155642 12429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:08.156199 12429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:08.156267 12429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:08.156317 12429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:08.156342 12429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:39:08 dist-test-slave-wp6g krb5kdc[12312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:08.163714 12429 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:46329
I20260620 18:39:08.163811 12560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:46329 every 8 connection(s)
I20260620 18:39:08.164146 12429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:08.167542 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.156766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:51727 (local address 127.10.250.126:46539)
0620 18:39:08.156884 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.156887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.157099 (+ 212us) server_negotiation.cc:407] Connection header received
0620 18:39:08.157736 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.157741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.157755 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.157832 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:08.158888 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.159473 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.160561 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.160772 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.160851 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.164064 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:08.164066 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:08.164068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:08.164083 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:08.165874 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:08.166369 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:08.166371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:08.166372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:08.166451 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:08.166780 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:08.166781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:08.166781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:08.166928 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:08.167023 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:08.167333 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 18:39:08.167435 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260620 18:39:08.167742 12454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.156768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46539 (local address 127.10.250.65:51727)
0620 18:39:08.156947 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.156990 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.157591 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.157963 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.157969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.158406 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:08.158768 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.158776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.159609 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.159610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.160444 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.160447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.160887 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.160889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.161014 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.161657 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:08.161668 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:08.163932 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:08.166000 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:08.166008 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:08.166011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:08.166264 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:08.166547 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:08.166548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:08.166548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:08.166685 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:08.167012 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:08.167020 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:08.167181 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 18:39:08.167296 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 18:39:08.167488 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
I20260620 18:39:08.168339 12561 heartbeater.cc:347] Connected to a master server at 127.10.250.126:46539
I20260620 18:39:08.168462 12561 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:08.168656 12561 heartbeater.cc:511] Master 127.10.250.126:46539 requested a full tablet report, sending...
I20260620 18:39:08.169265 12356 ts_manager.cc:194] Registered new tserver with Master: a3b3753324af4b47a13b9e2f99d4d669 (127.10.250.65:46329)
I20260620 18:39:08.170258 12356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:51727
I20260620 18:39:08.173434 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 12429
I20260620 18:39:08.173517 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:08.173717 11241 external_mini_cluster.cc:1428] Setting key af72f57afbec705087c90714eff38f85
I20260620 18:39:08.176460 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.169479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46539 (local address 127.10.250.65:60151)
0620 18:39:08.169668 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.169680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.169749 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.169875 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.169876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.170083 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:08.170155 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.170157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.171110 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.171112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.171724 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.171727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.172172 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.172173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.172234 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.172725 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:08.172729 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:08.173055 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:08.175249 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:08.175250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:08.175250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:08.175445 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:08.175693 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:08.175694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:08.175694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:08.175767 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:08.176047 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:08.176048 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:08.176125 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 18:39:08.176228 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:39:08.176344 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":107,"threads_started":1}
I20260620 18:39:08.176447 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.169572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:60151 (local address 127.10.250.126:46539)
0620 18:39:08.169624 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.169626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.169733 (+ 107us) server_negotiation.cc:407] Connection header received
0620 18:39:08.169806 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.169807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.169818 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.169870 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:08.170240 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.170933 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.171819 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.172087 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.172152 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.173140 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:08.173141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:08.173141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:08.173148 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:08.175135 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:08.175523 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:08.175524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:08.175524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:08.175590 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:08.175830 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:08.175831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:08.175832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:08.175982 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:08.176051 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:08.176260 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 18:39:08.176348 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:39:08.202704 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:46539
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:43173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:08.286008 12570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:08.286249 12570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:08.286278 12570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:08.287695 12570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:08.287750 12570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:08.287774 12570 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:39:08.287793 12570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:08.289340 12570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:43173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.250.126:46539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:08.289603 12570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:08.289873 12570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:08.295552 12576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:08.295490 12578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:08.295495 12575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:08.295811 12570 server_base.cc:1034] running on GCE node
I20260620 18:39:08.296029 12570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:08.296337 12570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:08.297497 12570 hybrid_clock.cc:648] HybridClock initialized: now 1781980748297478 us; error 38 us; skew 500 ppm
Jun 20 18:39:08 dist-test-slave-wp6g krb5kdc[12312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:08.300169 12570 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:39:08.300680 12570 webserver.cc:460] Webserver started at http://127.10.250.66:36771/ using document root <none> and password file <none>
I20260620 18:39:08.300942 12570 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:08.301012 12570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:08.301165 12570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:08.302304 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "0b2c3fe34fc84e20a74cab2131786b2c"
format_stamp: "Formatted at 2026-06-20 18:39:08 on dist-test-slave-wp6g"
server_key: "27104ab4e35fded8eb561d3248d87b38"
server_key_iv: "45e686576719a0f726fdd63b07966f98"
server_key_version: "encryptionkey@0"
I20260620 18:39:08.302718 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "0b2c3fe34fc84e20a74cab2131786b2c"
format_stamp: "Formatted at 2026-06-20 18:39:08 on dist-test-slave-wp6g"
server_key: "27104ab4e35fded8eb561d3248d87b38"
server_key_iv: "45e686576719a0f726fdd63b07966f98"
server_key_version: "encryptionkey@0"
I20260620 18:39:08.304190 12570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:08.305250 12585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:08.305503 12570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:08.305626 12570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "0b2c3fe34fc84e20a74cab2131786b2c"
format_stamp: "Formatted at 2026-06-20 18:39:08 on dist-test-slave-wp6g"
server_key: "27104ab4e35fded8eb561d3248d87b38"
server_key_iv: "45e686576719a0f726fdd63b07966f98"
server_key_version: "encryptionkey@0"
I20260620 18:39:08.305711 12570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:08.313325 12570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:08.315785 12570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:08.315966 12570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:08.316684 12570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:08.316735 12570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:08.316769 12570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:08.316802 12570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:39:08 dist-test-slave-wp6g krb5kdc[12312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:08.324407 12570 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:40247
I20260620 18:39:08.324512 12700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:40247 every 8 connection(s)
I20260620 18:39:08.324911 12570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:08.327284 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.317145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:48891 (local address 127.10.250.126:46539)
0620 18:39:08.317216 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.317219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.317462 (+ 243us) server_negotiation.cc:407] Connection header received
0620 18:39:08.318040 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.318041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.318057 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.318121 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:08.318912 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.319826 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.321026 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.321256 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.321376 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.323887 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:08.323889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:08.323890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:08.323900 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:08.325729 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:08.326156 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:08.326157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:08.326158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:08.326242 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:08.326503 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:08.326504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:08.326505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:08.326739 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:08.326809 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:08.327108 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 18:39:08.327217 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 18:39:08.327416 12594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.317128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46539 (local address 127.10.250.66:48891)
0620 18:39:08.317322 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.317365 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.317884 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.318227 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.318230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.318605 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:08.318825 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.318830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.319990 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.319991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.320886 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.320890 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.321404 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.321405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.321555 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.322160 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:08.322179 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:08.323766 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:08.325831 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:08.325837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:08.325839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:08.326077 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:08.326307 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:08.326307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:08.326307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:08.326419 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:08.326812 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:08.326817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:08.326949 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 18:39:08.327045 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:39:08.327197 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":119,"threads_started":1}
I20260620 18:39:08.328011 12701 heartbeater.cc:347] Connected to a master server at 127.10.250.126:46539
I20260620 18:39:08.328293 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 12570
I20260620 18:39:08.328364 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:08.328544 11241 external_mini_cluster.cc:1428] Setting key 0d3a609ec975f4f2c17c371862f25112
I20260620 18:39:08.328735 12701 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:08.329067 12701 heartbeater.cc:511] Master 127.10.250.126:46539 requested a full tablet report, sending...
I20260620 18:39:08.329521 12357 ts_manager.cc:194] Registered new tserver with Master: 0b2c3fe34fc84e20a74cab2131786b2c (127.10.250.66:40247)
I20260620 18:39:08.329928 12357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:48891
I20260620 18:39:08.336755 12707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.329198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46539 (local address 127.10.250.66:43067)
0620 18:39:08.329333 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.329345 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.329435 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.329628 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.329629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.329860 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:08.329953 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.329956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.331004 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.331005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.331786 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.331789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.332244 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.332245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.332318 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.332872 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:08.332876 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:08.333329 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:08.335406 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:08.335408 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:08.335408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:08.335712 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:08.335954 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:08.335955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:08.335955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:08.336015 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:08.336377 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:08.336378 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:08.336428 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 18:39:08.336529 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:39:08.336643 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260620 18:39:08.336745 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.329209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:43067 (local address 127.10.250.126:46539)
0620 18:39:08.329257 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.329259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.329406 (+ 147us) server_negotiation.cc:407] Connection header received
0620 18:39:08.329512 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.329513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.329524 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.329556 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:08.330074 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.330863 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.331890 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.332137 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.332241 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.333445 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:08.333446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:08.333446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:08.333456 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:08.335311 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:08.335810 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:08.335812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:08.335812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:08.335877 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:08.336113 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:08.336114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:08.336115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:08.336285 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:08.336384 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:08.336555 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 18:39:08.336640 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:39:08.358861 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:46539
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:43173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:08.444516 12710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:08.444741 12710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:08.444767 12710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:08.446102 12710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:08.446157 12710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:08.446182 12710 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:39:08.446230 12710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:08.447739 12710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:43173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.250.126:46539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:08.448019 12710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:08.448258 12710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:08.453383 12718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:08.453370 12716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:08.453370 12715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:08.453557 12710 server_base.cc:1034] running on GCE node
I20260620 18:39:08.453871 12710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:08.454154 12710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:08.455366 12710 hybrid_clock.cc:648] HybridClock initialized: now 1781980748455338 us; error 37 us; skew 500 ppm
Jun 20 18:39:08 dist-test-slave-wp6g krb5kdc[12312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:08.458002 12710 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:39:08.458473 12710 webserver.cc:460] Webserver started at http://127.10.250.67:32773/ using document root <none> and password file <none>
I20260620 18:39:08.458765 12710 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:08.458812 12710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:08.458972 12710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:08.460039 12710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "68c905e548394ea380dfaeba94318fc6"
format_stamp: "Formatted at 2026-06-20 18:39:08 on dist-test-slave-wp6g"
server_key: "497ecdf3a75600f8e0bd5bd84d258e35"
server_key_iv: "281db100568c433e737423ce7ab4ed10"
server_key_version: "encryptionkey@0"
I20260620 18:39:08.460407 12710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "68c905e548394ea380dfaeba94318fc6"
format_stamp: "Formatted at 2026-06-20 18:39:08 on dist-test-slave-wp6g"
server_key: "497ecdf3a75600f8e0bd5bd84d258e35"
server_key_iv: "281db100568c433e737423ce7ab4ed10"
server_key_version: "encryptionkey@0"
I20260620 18:39:08.461784 12710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:08.462734 12725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:08.462918 12710 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:08.463016 12710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "68c905e548394ea380dfaeba94318fc6"
format_stamp: "Formatted at 2026-06-20 18:39:08 on dist-test-slave-wp6g"
server_key: "497ecdf3a75600f8e0bd5bd84d258e35"
server_key_iv: "281db100568c433e737423ce7ab4ed10"
server_key_version: "encryptionkey@0"
I20260620 18:39:08.463076 12710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:08.472301 12710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:08.474423 12710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:08.474622 12710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:08.475148 12710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:08.475194 12710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:08.475219 12710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:08.475231 12710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:08.481880 12710 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:35095
I20260620 18:39:08.481971 12840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:35095 every 8 connection(s)
I20260620 18:39:08.482331 12710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
Jun 20 18:39:08 dist-test-slave-wp6g krb5kdc[12312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:08.484603 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 12710
I20260620 18:39:08.484692 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:08.484911 11241 external_mini_cluster.cc:1428] Setting key 6354e7d98d7c2ad2ca9771f2670fa41f
I20260620 18:39:08.486402 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.475779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:45003 (local address 127.10.250.126:46539)
0620 18:39:08.475823 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.475825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.476078 (+ 253us) server_negotiation.cc:407] Connection header received
0620 18:39:08.476691 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.476693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.476711 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.476823 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:08.477754 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.478617 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.480195 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.480546 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.480628 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.483452 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:08.483454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:08.483454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:08.483461 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:08.484834 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:08.485319 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:08.485321 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:08.485321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:08.485385 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:08.485670 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:08.485671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:08.485672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:08.485805 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:08.485887 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:08.486219 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 18:39:08.486305 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:39:08.486778 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.475780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46539 (local address 127.10.250.67:45003)
0620 18:39:08.475955 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.475992 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.476509 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.476942 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.476946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.477340 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:08.477639 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.477645 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.478727 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.478728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.480099 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.480101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.480633 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.480634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.480790 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.481450 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:08.481458 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:08.483360 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:08.484909 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:08.484917 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:08.484920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:08.485227 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:08.485451 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:08.485452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:08.485452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:08.485615 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:08.485878 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:08.485884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:08.486070 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 18:39:08.486197 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 18:39:08.486376 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260620 18:39:08.487331 12841 heartbeater.cc:347] Connected to a master server at 127.10.250.126:46539
I20260620 18:39:08.487418 12841 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:08.487612 12841 heartbeater.cc:511] Master 127.10.250.126:46539 requested a full tablet report, sending...
I20260620 18:39:08.488025 12356 ts_manager.cc:194] Registered new tserver with Master: 68c905e548394ea380dfaeba94318fc6 (127.10.250.67:35095)
I20260620 18:39:08.488519 12356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:45003
I20260620 18:39:08.493973 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.488534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:56099 (local address 127.10.250.126:46539)
0620 18:39:08.488577 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.488579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.488642 (+ 63us) server_negotiation.cc:407] Connection header received
0620 18:39:08.488737 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.488738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.488748 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.488810 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:08.489169 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.489714 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.490441 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.490702 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.490758 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.491620 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:08.491621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:08.491621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:08.491628 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:08.493053 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:08.493388 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:08.493389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:08.493389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:08.493440 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:08.493569 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:08.493569 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:08.493570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:08.493665 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:08.493731 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:08.493846 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 18:39:08.493905 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:39:08.493984 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.488410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46539 (local address 127.10.250.67:56099)
0620 18:39:08.488582 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.488593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.488667 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.488818 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.488819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.489026 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:08.489088 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.489090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.489815 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.489816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.490371 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.490373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.490789 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.490790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.490857 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.491262 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:08.491265 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:08.491544 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:08.493140 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:08.493141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:08.493142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:08.493324 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:08.493480 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:08.493480 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:08.493481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:08.493524 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:08.493738 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:08.493739 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:08.493778 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 18:39:08.493830 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 18:39:08.493909 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":99,"threads_started":1}
I20260620 18:39:08.497107 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:39:08 dist-test-slave-wp6g krb5kdc[12312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980748, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:39:08 dist-test-slave-wp6g krb5kdc[12312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980748, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:08.521283 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.514272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.10.250.126:46539)
0620 18:39:08.514331 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.514335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.514353 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:39:08.514415 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.514415 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.514424 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.514485 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:08.515163 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.515663 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.516409 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.516589 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.516640 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.518986 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:08.518987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:08.518987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:08.518995 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:08.520295 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:08.520630 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:08.520631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:08.520631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:08.520683 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:08.520850 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:08.520850 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:08.520851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:08.520951 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:08.521027 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:08.521125 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 18:39:08.521214 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:39:08.523020 12356 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:08.524564 12356 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:08.531574 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.527465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:40247 (local address 127.0.0.1:47566)
0620 18:39:08.527752 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.527762 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.527818 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.528099 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.528100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.528105 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:08.528176 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.528179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.529440 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.529440 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.530492 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.530494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.531350 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.531351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.531419 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.531424 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:39:08.531469 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 18:39:08.531499 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":6,"thread_start_us":59,"threads_started":1}
I20260620 18:39:08.531705 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.527687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44536 (local address 127.10.250.65:46329)
0620 18:39:08.527901 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.527906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.527914 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:39:08.527948 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.527951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.527966 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.528034 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:08.528283 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.529077 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.530624 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.531123 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.531224 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.531288 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:08.531541 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 18:39:08.531581 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
I20260620 18:39:08.531746 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.527166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60006 (local address 127.10.250.67:35095)
0620 18:39:08.527370 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.527375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.527388 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:39:08.527431 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.527434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.527454 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.527535 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:08.527919 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.529093 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.530424 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.531248 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.531327 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.531385 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:08.531592 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 18:39:08.531634 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":123,"threads_started":1}
I20260620 18:39:08.531800 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.527309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:46329 (local address 127.0.0.1:44536)
0620 18:39:08.527421 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.527432 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.527488 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.528100 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.528100 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.528102 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:08.528176 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.528178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.529185 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.529187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.530484 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.530486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.531223 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.531224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.531311 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.531326 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 18:39:08.531395 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 18:39:08.531453 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":69,"threads_started":1}
I20260620 18:39:08.531996 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.527058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:35095 (local address 127.0.0.1:60006)
0620 18:39:08.527216 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.527257 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.527319 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.527615 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.527618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.527621 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:08.527717 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.527719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.529199 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.529200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.530326 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.530329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.531424 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.531425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.531502 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.531506 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:39:08.531854 (+ 348us) client_negotiation.cc:240] Negotiation successful
0620 18:39:08.531899 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":34,"thread_start_us":105,"threads_started":1}
I20260620 18:39:08.532186 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.527580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.10.250.66:40247)
0620 18:39:08.527742 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.527747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.527873 (+ 126us) server_negotiation.cc:407] Connection header received
0620 18:39:08.527916 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.527919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.527935 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.528007 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:08.528268 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.529358 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.530608 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.531286 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.531353 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.531401 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:08.531496 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 18:39:08.531549 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260620 18:39:08.532589 12495 tablet_service.cc:1467] Processing CreateTablet for tablet 0dbcd706f3da4d559c4291077de3d08e (DEFAULT_TABLE table=test-table [id=fcae1c925545490996d62b6fcf2a83a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:08.532819 12635 tablet_service.cc:1467] Processing CreateTablet for tablet 0dbcd706f3da4d559c4291077de3d08e (DEFAULT_TABLE table=test-table [id=fcae1c925545490996d62b6fcf2a83a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:08.532992 12495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dbcd706f3da4d559c4291077de3d08e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:08.533111 12635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dbcd706f3da4d559c4291077de3d08e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:08.535465 12775 tablet_service.cc:1467] Processing CreateTablet for tablet 0dbcd706f3da4d559c4291077de3d08e (DEFAULT_TABLE table=test-table [id=fcae1c925545490996d62b6fcf2a83a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:08.535810 12775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dbcd706f3da4d559c4291077de3d08e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:08.537201 12865 tablet_bootstrap.cc:492] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c: Bootstrap starting.
I20260620 18:39:08.537865 12865 tablet_bootstrap.cc:654] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:08.538206 12865 log.cc:824] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:08.538622 12866 tablet_bootstrap.cc:492] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669: Bootstrap starting.
I20260620 18:39:08.538977 12868 tablet_bootstrap.cc:492] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6: Bootstrap starting.
I20260620 18:39:08.539106 12865 tablet_bootstrap.cc:492] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c: No bootstrap required, opened a new log
I20260620 18:39:08.539178 12865 ts_tablet_manager.cc:1397] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:39:08.539345 12866 tablet_bootstrap.cc:654] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:08.539659 12866 log.cc:824] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:08.540063 12868 tablet_bootstrap.cc:654] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:08.540434 12868 log.cc:824] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:08.540623 12866 tablet_bootstrap.cc:492] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669: No bootstrap required, opened a new log
I20260620 18:39:08.540717 12866 ts_tablet_manager.cc:1397] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:39:08.541311 12868 tablet_bootstrap.cc:492] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6: No bootstrap required, opened a new log
I20260620 18:39:08.541388 12868 ts_tablet_manager.cc:1397] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:39:08.541623 12865 raft_consensus.cc:348] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } }
I20260620 18:39:08.541816 12865 raft_consensus.cc:374] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:08.541849 12865 raft_consensus.cc:729] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b2c3fe34fc84e20a74cab2131786b2c, State: Initialized, Role: FOLLOWER
I20260620 18:39:08.541985 12865 consensus_queue.cc:260] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } }
I20260620 18:39:08.542266 12865 ts_tablet_manager.cc:1428] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:08.542281 12701 heartbeater.cc:503] Master 127.10.250.126:46539 was elected leader, sending a full tablet report...
I20260620 18:39:08.542840 12866 raft_consensus.cc:348] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } }
I20260620 18:39:08.542986 12866 raft_consensus.cc:374] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:08.543028 12866 raft_consensus.cc:729] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b3753324af4b47a13b9e2f99d4d669, State: Initialized, Role: FOLLOWER
I20260620 18:39:08.543138 12866 consensus_queue.cc:260] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } }
I20260620 18:39:08.543423 12866 ts_tablet_manager.cc:1428] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 18:39:08.543572 12561 heartbeater.cc:503] Master 127.10.250.126:46539 was elected leader, sending a full tablet report...
I20260620 18:39:08.543501 12868 raft_consensus.cc:348] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } }
I20260620 18:39:08.543648 12868 raft_consensus.cc:374] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:08.543684 12868 raft_consensus.cc:729] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c905e548394ea380dfaeba94318fc6, State: Initialized, Role: FOLLOWER
I20260620 18:39:08.543792 12868 consensus_queue.cc:260] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } }
I20260620 18:39:08.544047 12868 ts_tablet_manager.cc:1428] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:08.544175 12841 heartbeater.cc:503] Master 127.10.250.126:46539 was elected leader, sending a full tablet report...
W20260620 18:39:08.575614 12702 tablet.cc:2369] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:39:08.665331 12562 tablet.cc:2369] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:39:08.674727 12873 raft_consensus.cc:482] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:08.674896 12873 raft_consensus.cc:504] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } }
I20260620 18:39:08.675298 12873 leader_election.cc:288] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3b3753324af4b47a13b9e2f99d4d669 (127.10.250.65:46329), 0b2c3fe34fc84e20a74cab2131786b2c (127.10.250.66:40247)
I20260620 18:39:08.678593 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.675699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:41917 (local address 127.10.250.65:46329)
0620 18:39:08.675757 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.675759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.675770 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:39:08.675857 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.675858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.675868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.675927 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:08.676151 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.676781 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.677853 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.678224 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.678307 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.678316 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:08.678472 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 18:39:08.678528 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 18:39:08.678599 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.675658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:46329 (local address 127.10.250.67:41917)
0620 18:39:08.675721 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.675735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.675779 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.675937 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.675938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.675940 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:08.676063 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.676066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.676860 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.676861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.677764 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.677767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.678315 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.678316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.678397 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.678400 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:08.678466 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 18:39:08.678504 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":52}
I20260620 18:39:08.678776 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.675894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:41301 (local address 127.10.250.66:40247)
0620 18:39:08.675951 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.675953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.676029 (+ 76us) server_negotiation.cc:407] Connection header received
0620 18:39:08.676073 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.676074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.676084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.676137 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:08.676282 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.676797 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.677867 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.678246 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.678336 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.678344 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:08.678636 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 18:39:08.678684 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:39:08.678891 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.675765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:40247 (local address 127.10.250.67:41301)
0620 18:39:08.675953 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.675963 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.676017 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.676143 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.676144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.676145 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:08.676215 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.676216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.676892 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.676892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.677769 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.677772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.678450 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.678451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.678562 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.678565 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:08.678626 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 18:39:08.678793 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":117,"threads_started":1}
I20260620 18:39:08.679054 12515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbcd706f3da4d559c4291077de3d08e" candidate_uuid: "68c905e548394ea380dfaeba94318fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3b3753324af4b47a13b9e2f99d4d669" is_pre_election: true
I20260620 18:39:08.679245 12515 raft_consensus.cc:2454] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68c905e548394ea380dfaeba94318fc6 in term 0.
I20260620 18:39:08.679486 12655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbcd706f3da4d559c4291077de3d08e" candidate_uuid: "68c905e548394ea380dfaeba94318fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" is_pre_election: true
I20260620 18:39:08.679514 12726 leader_election.cc:302] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68c905e548394ea380dfaeba94318fc6, a3b3753324af4b47a13b9e2f99d4d669; no voters:
I20260620 18:39:08.679637 12655 raft_consensus.cc:2454] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68c905e548394ea380dfaeba94318fc6 in term 0.
I20260620 18:39:08.679674 12873 raft_consensus.cc:2781] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:08.679736 12873 raft_consensus.cc:482] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:08.679766 12873 raft_consensus.cc:3037] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:08.680478 12873 raft_consensus.cc:504] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } }
I20260620 18:39:08.680626 12873 leader_election.cc:288] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [CANDIDATE]: Term 1 election: Requested vote from peers a3b3753324af4b47a13b9e2f99d4d669 (127.10.250.65:46329), 0b2c3fe34fc84e20a74cab2131786b2c (127.10.250.66:40247)
I20260620 18:39:08.680835 12515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbcd706f3da4d559c4291077de3d08e" candidate_uuid: "68c905e548394ea380dfaeba94318fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3b3753324af4b47a13b9e2f99d4d669"
I20260620 18:39:08.680876 12655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbcd706f3da4d559c4291077de3d08e" candidate_uuid: "68c905e548394ea380dfaeba94318fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b2c3fe34fc84e20a74cab2131786b2c"
I20260620 18:39:08.680935 12515 raft_consensus.cc:3037] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:08.680970 12655 raft_consensus.cc:3037] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:08.681660 12515 raft_consensus.cc:2454] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68c905e548394ea380dfaeba94318fc6 in term 1.
I20260620 18:39:08.681788 12655 raft_consensus.cc:2454] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68c905e548394ea380dfaeba94318fc6 in term 1.
I20260620 18:39:08.681867 12726 leader_election.cc:302] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68c905e548394ea380dfaeba94318fc6, a3b3753324af4b47a13b9e2f99d4d669; no voters:
I20260620 18:39:08.681943 12873 raft_consensus.cc:2781] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:08.682082 12873 raft_consensus.cc:686] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [term 1 LEADER]: Becoming Leader. State: Replica: 68c905e548394ea380dfaeba94318fc6, State: Running, Role: LEADER
I20260620 18:39:08.682190 12873 consensus_queue.cc:237] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } }
I20260620 18:39:08.683077 12356 catalog_manager.cc:5495] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68c905e548394ea380dfaeba94318fc6 (127.10.250.67). New cstate: current_term: 1 leader_uuid: "68c905e548394ea380dfaeba94318fc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } health_report { overall_health: UNKNOWN } } }
I20260620 18:39:08.717571 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.714808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60018 (local address 127.10.250.67:35095)
0620 18:39:08.714879 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.714881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.714890 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:08.714934 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.714937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.714960 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.715045 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:08.715232 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.715706 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.716513 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.716717 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.716794 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.717007 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:08.717113 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:08.717213 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:08.717254 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 18:39:08.717280 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:39:08.719494 12655 raft_consensus.cc:1261] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c [term 1 FOLLOWER]: Refusing update from remote peer 68c905e548394ea380dfaeba94318fc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:08.719519 12515 raft_consensus.cc:1261] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669 [term 1 FOLLOWER]: Refusing update from remote peer 68c905e548394ea380dfaeba94318fc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:08.719969 12873 consensus_queue.cc:1035] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:08.720173 12875 consensus_queue.cc:1035] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:08.722834 12878 mvcc.cc:205] Tried to move back new op lower bound from 7298993146752090112 to 7298993146603003904. Current Snapshot: MvccSnapshot[applied={T|T < 7298993146752090112}]
I20260620 18:39:08.723213 12880 mvcc.cc:205] Tried to move back new op lower bound from 7298993146752090112 to 7298993146603003904. Current Snapshot: MvccSnapshot[applied={T|T < 7298993146752090112}]
I20260620 18:39:08.723552 12879 mvcc.cc:205] Tried to move back new op lower bound from 7298993146752090112 to 7298993146603003904. Current Snapshot: MvccSnapshot[applied={T|T < 7298993146752090112}]
I20260620 18:39:08.727257 12355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:08.727371 12355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:08.728225 12355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f: Sending DeleteTablet for 3 replicas of tablet 0dbcd706f3da4d559c4291077de3d08e
I20260620 18:39:08.728554 12775 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dbcd706f3da4d559c4291077de3d08e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:08 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:60006
I20260620 18:39:08.728647 12495 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dbcd706f3da4d559c4291077de3d08e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:08 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:44536
I20260620 18:39:08.728658 12635 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dbcd706f3da4d559c4291077de3d08e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:08 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:47566
I20260620 18:39:08.728806 12888 tablet_replica.cc:330] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6: stopping tablet replica
I20260620 18:39:08.728904 12888 raft_consensus.cc:2229] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:08.729024 12888 raft_consensus.cc:2258] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:08.729176 12355 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:08.729326 12355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:08.729471 12888 ts_tablet_manager.cc:1905] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:08.729813 12890 tablet_replica.cc:330] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c: stopping tablet replica
I20260620 18:39:08.729928 12890 raft_consensus.cc:2229] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:08.730043 12890 raft_consensus.cc:2258] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:08.730212 12889 tablet_replica.cc:330] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669: stopping tablet replica
I20260620 18:39:08.730317 12889 raft_consensus.cc:2229] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:08.730423 12889 raft_consensus.cc:2258] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:08.730540 12890 ts_tablet_manager.cc:1905] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:08.730883 12775 tablet_service.cc:1467] Processing CreateTablet for tablet db679e42fff64ab1ab89f03deec9b208 (DEFAULT_TABLE table=test-table [id=25cc2ca09f314d9bbc7ab6e2526344ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:08.730865 12635 tablet_service.cc:1467] Processing CreateTablet for tablet db679e42fff64ab1ab89f03deec9b208 (DEFAULT_TABLE table=test-table [id=25cc2ca09f314d9bbc7ab6e2526344ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:08.730911 12889 ts_tablet_manager.cc:1905] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:08.730983 12775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db679e42fff64ab1ab89f03deec9b208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:08.730983 12635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db679e42fff64ab1ab89f03deec9b208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:08.732239 12868 tablet_bootstrap.cc:492] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6: Bootstrap starting.
I20260620 18:39:08.732352 12865 tablet_bootstrap.cc:492] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c: Bootstrap starting.
I20260620 18:39:08.732688 12868 tablet_bootstrap.cc:654] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:08.732707 12890 ts_tablet_manager.cc:1918] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:08.732774 12890 log.cc:1196] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/0dbcd706f3da4d559c4291077de3d08e
I20260620 18:39:08.732771 12865 tablet_bootstrap.cc:654] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:08.733047 12890 ts_tablet_manager.cc:1939] T 0dbcd706f3da4d559c4291077de3d08e P 0b2c3fe34fc84e20a74cab2131786b2c: Deleting consensus metadata
I20260620 18:39:08.733285 12495 tablet_service.cc:1467] Processing CreateTablet for tablet db679e42fff64ab1ab89f03deec9b208 (DEFAULT_TABLE table=test-table [id=25cc2ca09f314d9bbc7ab6e2526344ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:08.733464 12341 catalog_manager.cc:4841] TS 0b2c3fe34fc84e20a74cab2131786b2c (127.10.250.66:40247): tablet 0dbcd706f3da4d559c4291077de3d08e (table test-table [id=fcae1c925545490996d62b6fcf2a83a3]) successfully deleted
I20260620 18:39:08.734053 12865 tablet_bootstrap.cc:492] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c: No bootstrap required, opened a new log
I20260620 18:39:08.734117 12865 ts_tablet_manager.cc:1397] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:39:08.734264 12865 raft_consensus.cc:348] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.734362 12865 raft_consensus.cc:374] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:08.734392 12865 raft_consensus.cc:729] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b2c3fe34fc84e20a74cab2131786b2c, State: Initialized, Role: FOLLOWER
I20260620 18:39:08.734436 12865 consensus_queue.cc:260] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.734540 12865 ts_tablet_manager.cc:1428] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:08.734722 12888 ts_tablet_manager.cc:1918] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:08.734799 12868 tablet_bootstrap.cc:492] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6: No bootstrap required, opened a new log
I20260620 18:39:08.734881 12868 ts_tablet_manager.cc:1397] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 18:39:08.735038 12888 log.cc:1196] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/0dbcd706f3da4d559c4291077de3d08e
I20260620 18:39:08.735057 12868 raft_consensus.cc:348] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.735131 12868 raft_consensus.cc:374] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:08.735158 12868 raft_consensus.cc:729] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c905e548394ea380dfaeba94318fc6, State: Initialized, Role: FOLLOWER
I20260620 18:39:08.735220 12868 consensus_queue.cc:260] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.735342 12868 ts_tablet_manager.cc:1428] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:08.735486 12888 ts_tablet_manager.cc:1939] T 0dbcd706f3da4d559c4291077de3d08e P 68c905e548394ea380dfaeba94318fc6: Deleting consensus metadata
I20260620 18:39:08.735906 12341 catalog_manager.cc:4841] TS 68c905e548394ea380dfaeba94318fc6 (127.10.250.67:35095): tablet 0dbcd706f3da4d559c4291077de3d08e (table test-table [id=fcae1c925545490996d62b6fcf2a83a3]) successfully deleted
I20260620 18:39:08.738718 12495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db679e42fff64ab1ab89f03deec9b208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:08.738739 12889 ts_tablet_manager.cc:1918] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:08.738830 12889 log.cc:1196] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/wal/wals/0dbcd706f3da4d559c4291077de3d08e
I20260620 18:39:08.739107 12889 ts_tablet_manager.cc:1939] T 0dbcd706f3da4d559c4291077de3d08e P a3b3753324af4b47a13b9e2f99d4d669: Deleting consensus metadata
I20260620 18:39:08.740118 12866 tablet_bootstrap.cc:492] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669: Bootstrap starting.
I20260620 18:39:08.740231 12341 catalog_manager.cc:4841] TS a3b3753324af4b47a13b9e2f99d4d669 (127.10.250.65:46329): tablet 0dbcd706f3da4d559c4291077de3d08e (table test-table [id=fcae1c925545490996d62b6fcf2a83a3]) successfully deleted
I20260620 18:39:08.740579 12866 tablet_bootstrap.cc:654] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:08.741245 12866 tablet_bootstrap.cc:492] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669: No bootstrap required, opened a new log
I20260620 18:39:08.741284 12866 ts_tablet_manager.cc:1397] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 18:39:08.741460 12866 raft_consensus.cc:348] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.741519 12866 raft_consensus.cc:374] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:08.741533 12866 raft_consensus.cc:729] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b3753324af4b47a13b9e2f99d4d669, State: Initialized, Role: FOLLOWER
I20260620 18:39:08.741595 12866 consensus_queue.cc:260] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.741961 12866 ts_tablet_manager.cc:1428] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 18:39:08.750772 12883 raft_consensus.cc:482] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:08.750910 12883 raft_consensus.cc:504] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.751173 12883 leader_election.cc:288] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3b3753324af4b47a13b9e2f99d4d669 (127.10.250.65:46329), 0b2c3fe34fc84e20a74cab2131786b2c (127.10.250.66:40247)
I20260620 18:39:08.751312 12515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db679e42fff64ab1ab89f03deec9b208" candidate_uuid: "68c905e548394ea380dfaeba94318fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3b3753324af4b47a13b9e2f99d4d669" is_pre_election: true
I20260620 18:39:08.751423 12515 raft_consensus.cc:2454] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68c905e548394ea380dfaeba94318fc6 in term 0.
I20260620 18:39:08.751379 12655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db679e42fff64ab1ab89f03deec9b208" candidate_uuid: "68c905e548394ea380dfaeba94318fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" is_pre_election: true
I20260620 18:39:08.751489 12655 raft_consensus.cc:2454] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68c905e548394ea380dfaeba94318fc6 in term 0.
I20260620 18:39:08.751583 12726 leader_election.cc:302] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68c905e548394ea380dfaeba94318fc6, a3b3753324af4b47a13b9e2f99d4d669; no voters:
I20260620 18:39:08.751685 12883 raft_consensus.cc:2781] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:08.751739 12883 raft_consensus.cc:482] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:08.751775 12883 raft_consensus.cc:3037] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:08.752604 12883 raft_consensus.cc:504] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.752758 12883 leader_election.cc:288] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [CANDIDATE]: Term 1 election: Requested vote from peers a3b3753324af4b47a13b9e2f99d4d669 (127.10.250.65:46329), 0b2c3fe34fc84e20a74cab2131786b2c (127.10.250.66:40247)
I20260620 18:39:08.752939 12655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db679e42fff64ab1ab89f03deec9b208" candidate_uuid: "68c905e548394ea380dfaeba94318fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b2c3fe34fc84e20a74cab2131786b2c"
I20260620 18:39:08.752980 12515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db679e42fff64ab1ab89f03deec9b208" candidate_uuid: "68c905e548394ea380dfaeba94318fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3b3753324af4b47a13b9e2f99d4d669"
I20260620 18:39:08.753048 12655 raft_consensus.cc:3037] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:08.753083 12515 raft_consensus.cc:3037] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:08.753742 12515 raft_consensus.cc:2454] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68c905e548394ea380dfaeba94318fc6 in term 1.
I20260620 18:39:08.753831 12655 raft_consensus.cc:2454] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68c905e548394ea380dfaeba94318fc6 in term 1.
I20260620 18:39:08.753962 12726 leader_election.cc:302] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68c905e548394ea380dfaeba94318fc6, a3b3753324af4b47a13b9e2f99d4d669; no voters:
I20260620 18:39:08.754106 12883 raft_consensus.cc:2781] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:08.754176 12883 raft_consensus.cc:686] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [term 1 LEADER]: Becoming Leader. State: Replica: 68c905e548394ea380dfaeba94318fc6, State: Running, Role: LEADER
I20260620 18:39:08.754230 12883 consensus_queue.cc:237] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.754846 12355 catalog_manager.cc:5495] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68c905e548394ea380dfaeba94318fc6 (127.10.250.67). New cstate: current_term: 1 leader_uuid: "68c905e548394ea380dfaeba94318fc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } health_report { overall_health: HEALTHY } } }
Jun 20 18:39:08 dist-test-slave-wp6g krb5kdc[12312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980746, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:08.795746 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.787453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46539 (local address 127.0.0.1:44084)
0620 18:39:08.787668 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.787697 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.787816 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.787985 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.787986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.788328 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:08.788391 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.788393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.789369 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.789371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.789937 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.789939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.790431 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.790432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.790474 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.791160 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:08.791164 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:08.792619 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:08.794604 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:08.794613 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:08.794616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:08.794849 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:08.795087 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:08.795088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:08.795088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:08.795144 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:08.795445 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:08.795452 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:08.795497 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:39:08.795557 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 18:39:08.795652 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":114,"threads_started":1}
I20260620 18:39:08.795843 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.787538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44084 (local address 127.10.250.126:46539)
0620 18:39:08.787605 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.787607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.787812 (+ 205us) server_negotiation.cc:407] Connection header received
0620 18:39:08.787868 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.787869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.787882 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.787973 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:08.788485 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.789251 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.790037 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.790337 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.790417 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.792726 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:08.792728 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:08.792728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:08.792738 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:08.794456 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:08.794929 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:08.794930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:08.794930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:08.795009 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:08.795209 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:08.795210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:08.795211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:08.795367 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:08.795455 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:08.795607 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 18:39:08.795722 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":38}
I20260620 18:39:08.798379 12355 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:44084:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:39:08.798647 12355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:08.800447 12635 tablet_service.cc:1467] Processing CreateTablet for tablet a66a2870b7b245c2bac549f5f9a3d808 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6045cba0e6254994a7dc576e8c0a38c4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:39:08.800480 12495 tablet_service.cc:1467] Processing CreateTablet for tablet a66a2870b7b245c2bac549f5f9a3d808 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6045cba0e6254994a7dc576e8c0a38c4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:39:08.800470 12775 tablet_service.cc:1467] Processing CreateTablet for tablet a66a2870b7b245c2bac549f5f9a3d808 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6045cba0e6254994a7dc576e8c0a38c4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:39:08.800619 12635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a66a2870b7b245c2bac549f5f9a3d808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:08.800623 12495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a66a2870b7b245c2bac549f5f9a3d808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:08.800621 12775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a66a2870b7b245c2bac549f5f9a3d808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:08.802107 12868 tablet_bootstrap.cc:492] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6: Bootstrap starting.
I20260620 18:39:08.802152 12866 tablet_bootstrap.cc:492] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669: Bootstrap starting.
I20260620 18:39:08.802435 12868 tablet_bootstrap.cc:654] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:08.802592 12865 tablet_bootstrap.cc:492] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c: Bootstrap starting.
I20260620 18:39:08.802621 12866 tablet_bootstrap.cc:654] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:08.803016 12865 tablet_bootstrap.cc:654] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:08.803223 12866 tablet_bootstrap.cc:492] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669: No bootstrap required, opened a new log
I20260620 18:39:08.803258 12866 ts_tablet_manager.cc:1397] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:08.803427 12866 raft_consensus.cc:348] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.803514 12866 raft_consensus.cc:374] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:08.803535 12866 raft_consensus.cc:729] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b3753324af4b47a13b9e2f99d4d669, State: Initialized, Role: FOLLOWER
I20260620 18:39:08.803608 12866 consensus_queue.cc:260] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.803735 12866 ts_tablet_manager.cc:1428] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 18:39:08.803808 12872 tablet_replica.cc:439] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } }
I20260620 18:39:08.803871 12872 tablet_replica.cc:442] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:08.803886 12865 tablet_bootstrap.cc:492] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c: No bootstrap required, opened a new log
I20260620 18:39:08.803931 12865 ts_tablet_manager.cc:1397] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:08.804095 12865 raft_consensus.cc:348] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.804154 12865 raft_consensus.cc:374] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:08.804172 12865 raft_consensus.cc:729] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b2c3fe34fc84e20a74cab2131786b2c, State: Initialized, Role: FOLLOWER
I20260620 18:39:08.804211 12865 consensus_queue.cc:260] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.804287 12865 ts_tablet_manager.cc:1428] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:08.804427 12871 tablet_replica.cc:439] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } }
I20260620 18:39:08.804503 12871 tablet_replica.cc:442] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:08.805186 12868 tablet_bootstrap.cc:492] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6: No bootstrap required, opened a new log
I20260620 18:39:08.805243 12868 ts_tablet_manager.cc:1397] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 18:39:08.805380 12868 raft_consensus.cc:348] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.805451 12868 raft_consensus.cc:374] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:08.805470 12868 raft_consensus.cc:729] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c905e548394ea380dfaeba94318fc6, State: Initialized, Role: FOLLOWER
I20260620 18:39:08.805509 12868 consensus_queue.cc:260] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.805590 12868 ts_tablet_manager.cc:1428] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 18:39:08.805701 12883 tablet_replica.cc:439] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } }
I20260620 18:39:08.805768 12883 tablet_replica.cc:442] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 18:39:08.983217 12842 tablet.cc:2369] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:39:08.984340 12872 raft_consensus.cc:482] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:08.984432 12872 raft_consensus.cc:504] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.984730 12872 leader_election.cc:288] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b2c3fe34fc84e20a74cab2131786b2c (127.10.250.66:40247), 68c905e548394ea380dfaeba94318fc6 (127.10.250.67:35095)
I20260620 18:39:08.988348 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.985398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:51161 (local address 127.10.250.67:35095)
0620 18:39:08.985445 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.985447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.985542 (+ 95us) server_negotiation.cc:407] Connection header received
0620 18:39:08.985601 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.985602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.985613 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.985668 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:08.985856 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.986303 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.987587 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.987963 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.988024 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.988032 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:08.988216 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 18:39:08.988266 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:39:08.988386 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.985198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:36425 (local address 127.10.250.66:40247)
0620 18:39:08.985252 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.985254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.985370 (+ 116us) server_negotiation.cc:407] Connection header received
0620 18:39:08.985487 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.985488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.985499 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.985530 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:08.985797 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.986296 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.987607 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.987980 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.988108 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.988118 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:08.988276 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 18:39:08.988324 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 18:39:08.988394 12905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.985341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:35095 (local address 127.10.250.65:51161)
0620 18:39:08.985483 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.985493 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.985545 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.985688 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.985689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.985690 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:08.985770 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.985772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.986417 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.986418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.987487 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.987490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.988062 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.988064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.988140 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.988144 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:39:08.988203 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 18:39:08.988261 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":61,"thread_start_us":69,"threads_started":1}
I20260620 18:39:08.988405 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.985042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:40247 (local address 127.10.250.65:36425)
0620 18:39:08.985295 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.985307 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.985402 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.985606 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.985607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.985609 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:08.985709 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.985711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.986391 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.986393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.987486 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.987488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.988104 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.988105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.988192 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.988197 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:39:08.988255 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 18:39:08.988292 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":41,"thread_start_us":164,"threads_started":1}
I20260620 18:39:08.988679 12795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66a2870b7b245c2bac549f5f9a3d808" candidate_uuid: "a3b3753324af4b47a13b9e2f99d4d669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68c905e548394ea380dfaeba94318fc6" is_pre_election: true
I20260620 18:39:08.988840 12795 raft_consensus.cc:2454] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3b3753324af4b47a13b9e2f99d4d669 in term 0.
I20260620 18:39:08.988857 12655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66a2870b7b245c2bac549f5f9a3d808" candidate_uuid: "a3b3753324af4b47a13b9e2f99d4d669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" is_pre_election: true
I20260620 18:39:08.988951 12655 raft_consensus.cc:2454] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3b3753324af4b47a13b9e2f99d4d669 in term 0.
I20260620 18:39:08.989063 12445 leader_election.cc:302] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68c905e548394ea380dfaeba94318fc6, a3b3753324af4b47a13b9e2f99d4d669; no voters:
I20260620 18:39:08.989197 12872 raft_consensus.cc:2781] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:08.989248 12872 raft_consensus.cc:482] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:08.989276 12872 raft_consensus.cc:3037] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:08.990374 12872 raft_consensus.cc:504] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.990561 12872 leader_election.cc:288] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [CANDIDATE]: Term 1 election: Requested vote from peers 0b2c3fe34fc84e20a74cab2131786b2c (127.10.250.66:40247), 68c905e548394ea380dfaeba94318fc6 (127.10.250.67:35095)
I20260620 18:39:08.990777 12795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66a2870b7b245c2bac549f5f9a3d808" candidate_uuid: "a3b3753324af4b47a13b9e2f99d4d669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68c905e548394ea380dfaeba94318fc6"
I20260620 18:39:08.990777 12655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66a2870b7b245c2bac549f5f9a3d808" candidate_uuid: "a3b3753324af4b47a13b9e2f99d4d669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b2c3fe34fc84e20a74cab2131786b2c"
I20260620 18:39:08.990861 12795 raft_consensus.cc:3037] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:08.990872 12655 raft_consensus.cc:3037] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:08.991675 12795 raft_consensus.cc:2454] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3b3753324af4b47a13b9e2f99d4d669 in term 1.
I20260620 18:39:08.991668 12655 raft_consensus.cc:2454] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3b3753324af4b47a13b9e2f99d4d669 in term 1.
I20260620 18:39:08.991827 12445 leader_election.cc:302] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68c905e548394ea380dfaeba94318fc6, a3b3753324af4b47a13b9e2f99d4d669; no voters:
I20260620 18:39:08.991910 12872 raft_consensus.cc:2781] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:08.992085 12872 raft_consensus.cc:686] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [term 1 LEADER]: Becoming Leader. State: Replica: a3b3753324af4b47a13b9e2f99d4d669, State: Running, Role: LEADER
I20260620 18:39:08.992146 12872 consensus_queue.cc:237] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } }
I20260620 18:39:08.992534 12906 tablet_replica.cc:439] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669: TxnStatusTablet state changed. Reason: New leader a3b3753324af4b47a13b9e2f99d4d669. Latest consensus state: current_term: 1 leader_uuid: "a3b3753324af4b47a13b9e2f99d4d669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } }
I20260620 18:39:08.992650 12906 tablet_replica.cc:442] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:39:08.992890 12355 catalog_manager.cc:5495] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3b3753324af4b47a13b9e2f99d4d669 (127.10.250.65). New cstate: current_term: 1 leader_uuid: "a3b3753324af4b47a13b9e2f99d4d669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } health_report { overall_health: UNKNOWN } } }
I20260620 18:39:08.993223 12908 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:39:08.993386 12908 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:39:09.035892 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:09.032836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.10.250.65:46329)
0620 18:39:09.032875 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:09.032877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:09.032902 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:39:09.033004 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:09.033005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:09.033022 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:09.033118 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:09.033258 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.033790 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.034944 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.035216 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.035296 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:09.035490 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:09.035607 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:09.035697 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:09.035753 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 18:39:09.035785 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 18:39:09.035908 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:09.032776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:46329 (local address 127.0.0.1:44544)
0620 18:39:09.032858 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:09.032870 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:09.032921 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:09.033142 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:09.033143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:09.033144 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:39:09.033201 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:09.033203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.033867 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.033868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:09.034860 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:09.034864 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.035287 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.035288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:09.035349 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:09.035370 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:39:09.035692 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:39:09.035700 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:39:09.035751 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 18:39:09.035805 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 18:39:09.037201 12655 raft_consensus.cc:1261] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c [term 1 FOLLOWER]: Refusing update from remote peer a3b3753324af4b47a13b9e2f99d4d669: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:09.037215 12795 raft_consensus.cc:1261] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6 [term 1 FOLLOWER]: Refusing update from remote peer a3b3753324af4b47a13b9e2f99d4d669: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:09.037470 12906 consensus_queue.cc:1035] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:09.037614 12872 consensus_queue.cc:1035] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:09.038072 12871 tablet_replica.cc:439] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c: TxnStatusTablet state changed. Reason: New leader a3b3753324af4b47a13b9e2f99d4d669. Latest consensus state: current_term: 1 leader_uuid: "a3b3753324af4b47a13b9e2f99d4d669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } }
I20260620 18:39:09.038169 12871 tablet_replica.cc:442] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:09.038821 12872 tablet_replica.cc:439] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3b3753324af4b47a13b9e2f99d4d669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } }
I20260620 18:39:09.038908 12872 tablet_replica.cc:442] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:39:09.039044 12871 tablet_replica.cc:439] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3b3753324af4b47a13b9e2f99d4d669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } }
I20260620 18:39:09.039122 12871 tablet_replica.cc:442] T a66a2870b7b245c2bac549f5f9a3d808 P 0b2c3fe34fc84e20a74cab2131786b2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:09.039094 12883 tablet_replica.cc:439] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6: TxnStatusTablet state changed. Reason: New leader a3b3753324af4b47a13b9e2f99d4d669. Latest consensus state: current_term: 1 leader_uuid: "a3b3753324af4b47a13b9e2f99d4d669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } }
I20260620 18:39:09.039187 12883 tablet_replica.cc:442] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:09.040409 12906 tablet_replica.cc:439] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3b3753324af4b47a13b9e2f99d4d669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } }
I20260620 18:39:09.040504 12906 tablet_replica.cc:442] T a66a2870b7b245c2bac549f5f9a3d808 P a3b3753324af4b47a13b9e2f99d4d669: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:39:09.040807 12883 tablet_replica.cc:439] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3b3753324af4b47a13b9e2f99d4d669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 } } peers { permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 } } peers { permanent_uuid: "68c905e548394ea380dfaeba94318fc6" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 35095 } } }
I20260620 18:39:09.040890 12883 tablet_replica.cc:442] T a66a2870b7b245c2bac549f5f9a3d808 P 68c905e548394ea380dfaeba94318fc6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:09.046101 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:09.043497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:42917 (local address 127.10.250.65:46329)
0620 18:39:09.043536 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:09.043539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:09.043589 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:39:09.043654 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:09.043655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:09.043663 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:09.043708 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:09.043821 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.044283 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.045435 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.045647 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.045710 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:09.045896 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:09.045924 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:09.045965 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:09.046024 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 18:39:09.046046 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:39:09.046154 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:09.043408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:46329 (local address 127.10.250.67:42917)
0620 18:39:09.043538 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:09.043549 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:09.043604 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:09.043712 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:09.043713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:09.043715 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:39:09.043766 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:09.043767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.044361 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.044362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:09.045319 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:09.045322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.045732 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.045733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:09.045793 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:09.045811 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:39:09.045968 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:39:09.045969 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 18:39:09.046019 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 18:39:09.046069 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260620 18:39:09.048398 12515 raft_consensus.cc:1261] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669 [term 1 FOLLOWER]: Refusing update from remote peer 68c905e548394ea380dfaeba94318fc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:09.048391 12655 raft_consensus.cc:1261] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c [term 1 FOLLOWER]: Refusing update from remote peer 68c905e548394ea380dfaeba94318fc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:09.048616 12875 consensus_queue.cc:1035] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3b3753324af4b47a13b9e2f99d4d669" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 46329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:09.048763 12875 consensus_queue.cc:1035] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b2c3fe34fc84e20a74cab2131786b2c" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:09.058248 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:09.054910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:50947 (local address 127.10.250.67:35095)
0620 18:39:09.054960 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:09.054962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:09.055162 (+ 200us) server_negotiation.cc:407] Connection header received
0620 18:39:09.055194 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:09.055195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:09.055207 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:09.055261 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:09.055455 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.056172 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.057361 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.057609 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.057728 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:09.057961 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:09.058007 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:09.058059 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:09.058149 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 18:39:09.058172 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 18:39:09.058475 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:09.054873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:35095 (local address 127.10.250.65:50947)
0620 18:39:09.055083 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:09.055094 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:09.055153 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:09.055265 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:09.055266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:09.055267 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:39:09.055324 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:09.055325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.056310 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.056311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:09.057285 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:09.057288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.057736 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.057737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:09.057796 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:09.057840 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:39:09.058084 (+ 244us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:39:09.058086 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:39:09.058344 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 18:39:09.058391 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260620 18:39:09.068360 12355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:09.068491 12355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:09.069224 12355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46c34b012a0449d0bf42244ee9cacc5f: Sending DeleteTablet for 3 replicas of tablet db679e42fff64ab1ab89f03deec9b208
I20260620 18:39:09.069465 12495 tablet_service.cc:1514] Processing DeleteTablet for tablet db679e42fff64ab1ab89f03deec9b208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:09 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:44536
I20260620 18:39:09.069516 12635 tablet_service.cc:1514] Processing DeleteTablet for tablet db679e42fff64ab1ab89f03deec9b208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:09 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:47566
I20260620 18:39:09.069563 12775 tablet_service.cc:1514] Processing DeleteTablet for tablet db679e42fff64ab1ab89f03deec9b208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:09 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:60006
I20260620 18:39:09.069650 12889 tablet_replica.cc:330] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669: stopping tablet replica
I20260620 18:39:09.069650 12890 tablet_replica.cc:330] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c: stopping tablet replica
I20260620 18:39:09.069713 12890 raft_consensus.cc:2229] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:09.069713 12889 raft_consensus.cc:2229] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:09.069754 12890 raft_consensus.cc:2258] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:09.069765 12889 raft_consensus.cc:2258] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:09.069845 12888 tablet_replica.cc:330] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6: stopping tablet replica
I20260620 18:39:09.069906 12888 raft_consensus.cc:2229] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:09.069974 12888 raft_consensus.cc:2258] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:09.070148 12889 ts_tablet_manager.cc:1905] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:09.070333 12890 ts_tablet_manager.cc:1905] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:09.070878 12888 ts_tablet_manager.cc:1905] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:09.072428 12889 ts_tablet_manager.cc:1918] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:09.072504 12889 log.cc:1196] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-0/wal/wals/db679e42fff64ab1ab89f03deec9b208
I20260620 18:39:09.072734 12889 ts_tablet_manager.cc:1939] T db679e42fff64ab1ab89f03deec9b208 P a3b3753324af4b47a13b9e2f99d4d669: Deleting consensus metadata
I20260620 18:39:09.073145 12341 catalog_manager.cc:4841] TS a3b3753324af4b47a13b9e2f99d4d669 (127.10.250.65:46329): tablet db679e42fff64ab1ab89f03deec9b208 (table test-table [id=25cc2ca09f314d9bbc7ab6e2526344ee]) successfully deleted
I20260620 18:39:09.073901 12888 ts_tablet_manager.cc:1918] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:09.074025 12888 log.cc:1196] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/db679e42fff64ab1ab89f03deec9b208
I20260620 18:39:09.074018 12890 ts_tablet_manager.cc:1918] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:09.074067 12890 log.cc:1196] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/db679e42fff64ab1ab89f03deec9b208
I20260620 18:39:09.074256 12888 ts_tablet_manager.cc:1939] T db679e42fff64ab1ab89f03deec9b208 P 68c905e548394ea380dfaeba94318fc6: Deleting consensus metadata
I20260620 18:39:09.074280 12890 ts_tablet_manager.cc:1939] T db679e42fff64ab1ab89f03deec9b208 P 0b2c3fe34fc84e20a74cab2131786b2c: Deleting consensus metadata
I20260620 18:39:09.074690 12341 catalog_manager.cc:4841] TS 68c905e548394ea380dfaeba94318fc6 (127.10.250.67:35095): tablet db679e42fff64ab1ab89f03deec9b208 (table test-table [id=25cc2ca09f314d9bbc7ab6e2526344ee]) successfully deleted
I20260620 18:39:09.074787 12341 catalog_manager.cc:4841] TS 0b2c3fe34fc84e20a74cab2131786b2c (127.10.250.66:40247): tablet db679e42fff64ab1ab89f03deec9b208 (table test-table [id=25cc2ca09f314d9bbc7ab6e2526344ee]) successfully deleted
Jun 20 18:39:09 dist-test-slave-wp6g krb5kdc[12312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980748, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.250.65@KRBTEST.COM
I20260620 18:39:09.084486 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:09.070608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.10.250.65:46329)
0620 18:39:09.070650 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:09.070652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:09.074929 (+ 4277us) server_negotiation.cc:407] Connection header received
0620 18:39:09.075013 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:09.075014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:09.075027 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:09.075073 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:09.075848 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.076477 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.077588 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.077775 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.077828 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:09.081363 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:09.081379 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:09.081380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:09.081387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:09.083182 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:09.083591 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:09.083595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:09.083596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:09.083678 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:09.083892 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:09.083893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:09.083893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:09.084048 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:09.084240 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:09.084279 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 18:39:09.084364 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 18:39:09.084867 12555 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44558
I20260620 18:39:09.093575 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:09.085928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44086 (local address 127.10.250.126:46539)
0620 18:39:09.085982 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:09.085985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:09.085997 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:39:09.086035 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:09.086036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:09.086046 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:09.086092 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:09.086678 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.087199 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.088397 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.088719 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.088791 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:09.090372 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:09.090374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:09.090374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:09.090382 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:09.092191 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:09.092619 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:09.092620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:09.092621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:09.092686 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:09.093000 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:09.093001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:09.093001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:09.093152 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:09.093252 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:09.093412 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 18:39:09.093501 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 18:39:09.093935 12355 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44086
I20260620 18:39:09.094666 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 12429
I20260620 18:39:09.100617 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 12570
I20260620 18:39:09.105576 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 12710
I20260620 18:39:09.111318 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 12325
2026-06-20T18:39:09Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6164 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:09 dist-test-slave-wp6g krb5kdc[12928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:09 dist-test-slave-wp6g krb5kdc[12928](info): set up 2 sockets
Jun 20 18:39:09 dist-test-slave-wp6g krb5kdc[12928](info): commencing operation
krb5kdc: starting...
W20260620 18:39:10.162336 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:39:10 dist-test-slave-wp6g krb5kdc[12928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:39:10.285354 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:42093
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:42093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:10.369051 12941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:10.369323 12941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:10.369364 12941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:10.370646 12941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:10.370713 12941 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:39:10.370736 12941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:10.370770 12941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:10.370796 12941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:10.370818 12941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:10.372382 12941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:42093
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:42093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:10.372673 12941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:10.372905 12941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:10.378120 12946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:10.378334 12947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:10.378373 12941 server_base.cc:1034] running on GCE node
W20260620 18:39:10.378396 12949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:10.378765 12941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:10.379143 12941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:10.380321 12941 hybrid_clock.cc:648] HybridClock initialized: now 1781980750380300 us; error 57 us; skew 500 ppm
Jun 20 18:39:10 dist-test-slave-wp6g krb5kdc[12928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980750, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:10.383096 12941 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:39:10.383518 12941 webserver.cc:460] Webserver started at http://127.10.250.126:33973/ using document root <none> and password file <none>
I20260620 18:39:10.383723 12941 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:10.383764 12941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:10.383872 12941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:10.384940 12941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "14fc285cb2da4c4ab0627bdfe9c21f1f"
format_stamp: "Formatted at 2026-06-20 18:39:10 on dist-test-slave-wp6g"
server_key: "e2e4c74ce8246151071a2f718d0007e7"
server_key_iv: "01da2eddb7ccce6ce3d519aa4d2089fc"
server_key_version: "encryptionkey@0"
I20260620 18:39:10.385281 12941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "14fc285cb2da4c4ab0627bdfe9c21f1f"
format_stamp: "Formatted at 2026-06-20 18:39:10 on dist-test-slave-wp6g"
server_key: "e2e4c74ce8246151071a2f718d0007e7"
server_key_iv: "01da2eddb7ccce6ce3d519aa4d2089fc"
server_key_version: "encryptionkey@0"
I20260620 18:39:10.386906 12941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:39:10.387840 12956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:10.388022 12941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:10.388125 12941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "14fc285cb2da4c4ab0627bdfe9c21f1f"
format_stamp: "Formatted at 2026-06-20 18:39:10 on dist-test-slave-wp6g"
server_key: "e2e4c74ce8246151071a2f718d0007e7"
server_key_iv: "01da2eddb7ccce6ce3d519aa4d2089fc"
server_key_version: "encryptionkey@0"
I20260620 18:39:10.388208 12941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:10.403738 12941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:10.406206 12941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:10.406379 12941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:10.411166 12941 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:42093
I20260620 18:39:10.411258 13018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:42093 every 8 connection(s)
I20260620 18:39:10.411634 12941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:10.412375 13019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:10.415449 13019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f: Bootstrap starting.
I20260620 18:39:10.416086 13019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:10.416395 13019 log.cc:824] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:10.417115 13019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f: No bootstrap required, opened a new log
I20260620 18:39:10.419324 13019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fc285cb2da4c4ab0627bdfe9c21f1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42093 } }
I20260620 18:39:10.419471 13019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:10.419493 13019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14fc285cb2da4c4ab0627bdfe9c21f1f, State: Initialized, Role: FOLLOWER
I20260620 18:39:10.419682 13019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fc285cb2da4c4ab0627bdfe9c21f1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42093 } }
I20260620 18:39:10.419746 13019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:10.419770 13019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:10.419840 13019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:10.420382 13019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fc285cb2da4c4ab0627bdfe9c21f1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42093 } }
I20260620 18:39:10.420447 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 12941
I20260620 18:39:10.420508 13019 leader_election.cc:302] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14fc285cb2da4c4ab0627bdfe9c21f1f; no voters:
I20260620 18:39:10.420567 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:10.420727 11241 external_mini_cluster.cc:1428] Setting key c8ceed66c20e4b7b2d30055ba72a2dcd
I20260620 18:39:10.420727 13019 leader_election.cc:288] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:10.420753 13022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:10.420872 13022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [term 1 LEADER]: Becoming Leader. State: Replica: 14fc285cb2da4c4ab0627bdfe9c21f1f, State: Running, Role: LEADER
I20260620 18:39:10.420957 13022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fc285cb2da4c4ab0627bdfe9c21f1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42093 } }
I20260620 18:39:10.421005 13019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:10.421290 13022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14fc285cb2da4c4ab0627bdfe9c21f1f. Latest consensus state: current_term: 1 leader_uuid: "14fc285cb2da4c4ab0627bdfe9c21f1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fc285cb2da4c4ab0627bdfe9c21f1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42093 } } }
I20260620 18:39:10.421356 13022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:10.421554 13029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:10.421528 13023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14fc285cb2da4c4ab0627bdfe9c21f1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fc285cb2da4c4ab0627bdfe9c21f1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42093 } } }
I20260620 18:39:10.421798 13023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:10.422135 13029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:10.423842 13029 catalog_manager.cc:1347] Generated new cluster ID: d0796136c22145c89b85e199298ce083
I20260620 18:39:10.423897 13029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:39:10 dist-test-slave-wp6g krb5kdc[12928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:10.433036 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.421265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.10.250.126:42093)
0620 18:39:10.421528 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:10.421538 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:10.421572 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:39:10.421692 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:10.421699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:10.421773 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:10.422913 (+ 1140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:10.423020 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.424450 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.425558 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.426176 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.426273 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.429177 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:10.429183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:10.429183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:10.429197 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:10.431317 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:10.431704 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:10.431708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:10.431709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:10.431837 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:10.432043 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:10.432044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:10.432045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:10.432402 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:10.432569 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:10.432649 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 18:39:10.432798 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":87,"threads_started":1}
I20260620 18:39:10.440243 13029 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:10.440958 13029 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:10.448863 13029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f: Generated new TSK 0
I20260620 18:39:10.449101 13029 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:39:10.478133 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:42093
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:10.560590 13045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:10.560843 13045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:10.560887 13045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:10.562098 13045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:10.562170 13045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:10.562206 13045 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:39:10.562232 13045 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:39:10.562269 13045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:10.563809 13045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.250.126:42093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:10.564119 13045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:10.564359 13045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:10.569643 13053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:10.569614 13051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:10.569622 13050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:10.569664 13045 server_base.cc:1034] running on GCE node
I20260620 18:39:10.570096 13045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:10.570359 13045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:10.571552 13045 hybrid_clock.cc:648] HybridClock initialized: now 1781980750571536 us; error 42 us; skew 500 ppm
Jun 20 18:39:10 dist-test-slave-wp6g krb5kdc[12928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980750, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:10.574762 13045 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:39:10.575254 13045 webserver.cc:460] Webserver started at http://127.10.250.65:45447/ using document root <none> and password file <none>
I20260620 18:39:10.575511 13045 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:10.575582 13045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:10.575737 13045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:10.576885 13045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "a41c569c07254088a3c061f18a978072"
format_stamp: "Formatted at 2026-06-20 18:39:10 on dist-test-slave-wp6g"
server_key: "2db8e3d226e265a2d55e34336cd8c1e8"
server_key_iv: "e843e4cb8712ae78dbce38620b66febf"
server_key_version: "encryptionkey@0"
I20260620 18:39:10.577273 13045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "a41c569c07254088a3c061f18a978072"
format_stamp: "Formatted at 2026-06-20 18:39:10 on dist-test-slave-wp6g"
server_key: "2db8e3d226e265a2d55e34336cd8c1e8"
server_key_iv: "e843e4cb8712ae78dbce38620b66febf"
server_key_version: "encryptionkey@0"
I20260620 18:39:10.578786 13045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 18:39:10.579671 13060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:10.579862 13045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:10.579970 13045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "a41c569c07254088a3c061f18a978072"
format_stamp: "Formatted at 2026-06-20 18:39:10 on dist-test-slave-wp6g"
server_key: "2db8e3d226e265a2d55e34336cd8c1e8"
server_key_iv: "e843e4cb8712ae78dbce38620b66febf"
server_key_version: "encryptionkey@0"
I20260620 18:39:10.580050 13045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:10.600489 13045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:10.602715 13045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:10.602871 13045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:10.603456 13045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:10.603520 13045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:10.603565 13045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:10.603616 13045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:10.610741 13045 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:42789
I20260620 18:39:10.610848 13175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:42789 every 8 connection(s)
I20260620 18:39:10.611231 13045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:39:10 dist-test-slave-wp6g krb5kdc[12928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980750, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:10.613653 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 13045
I20260620 18:39:10.613750 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:10.613948 11241 external_mini_cluster.cc:1428] Setting key 0792c9f80cc84f88ff741e1946f2ebc2
I20260620 18:39:10.614996 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.604072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:51197 (local address 127.10.250.126:42093)
0620 18:39:10.604135 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:10.604137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:10.604370 (+ 233us) server_negotiation.cc:407] Connection header received
0620 18:39:10.604928 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:10.604929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:10.604957 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:10.605025 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:10.606136 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.607091 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.608836 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.609047 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.609110 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.612098 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:10.612100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:10.612100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:10.612108 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:10.613375 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:10.613877 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:10.613878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:10.613879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:10.613946 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:10.614233 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:10.614235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:10.614235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:10.614370 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:10.614455 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:10.614802 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 18:39:10.614905 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:39:10.615236 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.604029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:42093 (local address 127.10.250.65:51197)
0620 18:39:10.604240 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:10.604276 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:10.604809 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:10.605227 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:10.605233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:10.605661 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:10.605999 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.606007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.607221 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.607223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.608737 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.608740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.609142 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.609144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.609319 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.610013 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:10.610026 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:10.612005 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:10.613490 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:10.613497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:10.613498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:10.613798 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:10.614030 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:10.614030 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:10.614031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:10.614153 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:10.614445 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:10.614452 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:10.614663 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 18:39:10.614779 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:39:10.614946 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":121,"threads_started":1}
I20260620 18:39:10.615837 13176 heartbeater.cc:347] Connected to a master server at 127.10.250.126:42093
I20260620 18:39:10.615929 13176 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:10.616142 13176 heartbeater.cc:511] Master 127.10.250.126:42093 requested a full tablet report, sending...
I20260620 18:39:10.616671 12972 ts_manager.cc:194] Registered new tserver with Master: a41c569c07254088a3c061f18a978072 (127.10.250.65:42789)
I20260620 18:39:10.617318 12972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:51197
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
I20260620 18:39:10.624953 13183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.618734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:42093 (local address 127.10.250.65:59157)
0620 18:39:10.618892 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:10.618905 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:10.618975 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:10.619103 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:10.619103 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:10.619334 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:10.619417 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.619419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.620074 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.620075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.620981 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.620984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.621337 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.621338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.621426 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.621980 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:10.621986 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:10.622408 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:10.623924 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:10.623925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:10.623926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:10.624133 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:10.624301 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:10.624302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:10.624302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:10.624355 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:10.624700 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:10.624701 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:10.624748 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 18:39:10.624802 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 18:39:10.624881 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260620 18:39:10.624967 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.618797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:59157 (local address 127.10.250.126:42093)
0620 18:39:10.618863 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:10.618865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:10.618956 (+ 91us) server_negotiation.cc:407] Connection header received
0620 18:39:10.619035 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:10.619036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:10.619044 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:10.619089 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:10.619496 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.619970 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.621060 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.621254 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.621310 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.622500 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:10.622501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:10.622501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:10.622508 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:10.623857 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:10.624197 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:10.624198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:10.624198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:10.624246 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:10.624490 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:10.624492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:10.624492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:10.624636 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:10.624697 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:10.624837 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 18:39:10.624903 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:39:10.641336 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:42093
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:10.723367 13185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:10.723563 13185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:10.723587 13185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:10.724792 13185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:10.724835 13185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:10.724856 13185 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:39:10.724869 13185 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:39:10.724886 13185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:10.726240 13185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.250.126:42093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:10.726418 13185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:10.726661 13185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:10.731475 13190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:10.731534 13193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:10.731489 13191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:10.731753 13185 server_base.cc:1034] running on GCE node
I20260620 18:39:10.731930 13185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:10.732177 13185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:10.733326 13185 hybrid_clock.cc:648] HybridClock initialized: now 1781980750733313 us; error 30 us; skew 500 ppm
Jun 20 18:39:10 dist-test-slave-wp6g krb5kdc[12928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980750, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:10.735782 13185 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:39:10.736248 13185 webserver.cc:460] Webserver started at http://127.10.250.66:45835/ using document root <none> and password file <none>
I20260620 18:39:10.736475 13185 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:10.736521 13185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:10.736688 13185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:10.737740 13185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "2794a8e11097434d99c6cf6fe061f780"
format_stamp: "Formatted at 2026-06-20 18:39:10 on dist-test-slave-wp6g"
server_key: "6ea232a36b4fd003904953ad1d027247"
server_key_iv: "5a3c6a927a707347228aa3f49e393874"
server_key_version: "encryptionkey@0"
I20260620 18:39:10.738106 13185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "2794a8e11097434d99c6cf6fe061f780"
format_stamp: "Formatted at 2026-06-20 18:39:10 on dist-test-slave-wp6g"
server_key: "6ea232a36b4fd003904953ad1d027247"
server_key_iv: "5a3c6a927a707347228aa3f49e393874"
server_key_version: "encryptionkey@0"
I20260620 18:39:10.739527 13185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:39:10.740263 13200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:10.740458 13185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:10.740550 13185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "2794a8e11097434d99c6cf6fe061f780"
format_stamp: "Formatted at 2026-06-20 18:39:10 on dist-test-slave-wp6g"
server_key: "6ea232a36b4fd003904953ad1d027247"
server_key_iv: "5a3c6a927a707347228aa3f49e393874"
server_key_version: "encryptionkey@0"
I20260620 18:39:10.740624 13185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:10.757388 13185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:10.759557 13185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:10.759711 13185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:10.760247 13185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:10.760284 13185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:10.760311 13185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:10.760362 13185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:39:10 dist-test-slave-wp6g krb5kdc[12928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980750, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:10.768208 13185 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:39027
I20260620 18:39:10.768292 13315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:39027 every 8 connection(s)
I20260620 18:39:10.768597 13185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:10.770685 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.760885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:36631 (local address 127.10.250.126:42093)
0620 18:39:10.760944 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:10.760946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:10.761252 (+ 306us) server_negotiation.cc:407] Connection header received
0620 18:39:10.761840 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:10.761842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:10.761855 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:10.761917 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:10.762845 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.763420 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.764579 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.764792 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.764877 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.767581 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:10.767583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:10.767583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:10.767592 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:10.769218 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:10.769680 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:10.769682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:10.769682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:10.769744 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:10.770001 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:10.770003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:10.770003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:10.770131 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:10.770212 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:10.770489 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 18:39:10.770587 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 18:39:10.770900 13209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.760898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:42093 (local address 127.10.250.66:36631)
0620 18:39:10.761093 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:10.761164 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:10.761728 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:10.762048 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:10.762052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:10.762481 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:10.762750 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.762755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.763513 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.763514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.764483 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.764485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.764901 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.764902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.765029 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.765574 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:10.765587 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:10.767464 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:10.769319 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:10.769325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:10.769328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:10.769580 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:10.769812 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:10.769812 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:10.769813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:10.769931 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:10.770201 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:10.770207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:10.770354 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 18:39:10.770467 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:39:10.770664 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20260620 18:39:10.771399 13316 heartbeater.cc:347] Connected to a master server at 127.10.250.126:42093
I20260620 18:39:10.771486 13316 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:10.771682 13316 heartbeater.cc:511] Master 127.10.250.126:42093 requested a full tablet report, sending...
I20260620 18:39:10.772089 12972 ts_manager.cc:194] Registered new tserver with Master: 2794a8e11097434d99c6cf6fe061f780 (127.10.250.66:39027)
I20260620 18:39:10.772586 12972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:36631
I20260620 18:39:10.776419 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 13185
I20260620 18:39:10.776513 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:10.776711 11241 external_mini_cluster.cc:1428] Setting key 448818894165fa29ba6379873728586d
I20260620 18:39:10.778367 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.772506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:40619 (local address 127.10.250.126:42093)
0620 18:39:10.772549 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:10.772551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:10.772620 (+ 69us) server_negotiation.cc:407] Connection header received
0620 18:39:10.772697 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:10.772698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:10.772707 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:10.772775 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:10.773298 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.773805 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.774658 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.774840 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.774897 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.775782 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:10.775783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:10.775783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:10.775789 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:10.777235 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:10.777645 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:10.777646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:10.777646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:10.777701 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:10.777901 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:10.777902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:10.777902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:10.778035 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:10.778086 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:10.778249 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 18:39:10.778307 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:39:10.778541 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.772444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:42093 (local address 127.10.250.66:40619)
0620 18:39:10.772559 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:10.772569 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:10.772631 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:10.772932 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:10.772933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:10.773136 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:10.773217 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.773218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.773926 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.773927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.774570 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.774572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.774907 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.774908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.774964 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.775356 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:10.775361 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:10.775707 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:10.777315 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:10.777316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:10.777316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:10.777570 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:10.777768 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:10.777768 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:10.777769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:10.777840 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:10.778101 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:10.778101 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:10.778153 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 18:39:10.778233 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 18:39:10.778351 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:39:10.803035 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:42093
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:10.886152 13325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:10.886409 13325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:10.886453 13325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:10.887639 13325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:10.887713 13325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:10.887750 13325 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:39:10.887773 13325 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:39:10.887815 13325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:10.889281 13325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.250.126:42093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:10.889559 13325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:10.889807 13325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:39:10.895115 13325 server_base.cc:1034] running on GCE node
W20260620 18:39:10.895074 13330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:10.895080 13331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:10.895080 13333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:10.895539 13325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:10.895797 13325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:10.896950 13325 hybrid_clock.cc:648] HybridClock initialized: now 1781980750896922 us; error 60 us; skew 500 ppm
Jun 20 18:39:10 dist-test-slave-wp6g krb5kdc[12928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980750, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:10.899430 13325 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:39:10.899869 13325 webserver.cc:460] Webserver started at http://127.10.250.67:41347/ using document root <none> and password file <none>
I20260620 18:39:10.900096 13325 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:10.900164 13325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:10.900319 13325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:10.901386 13325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "44c5a2ce2b1148389b55f7570c243f01"
format_stamp: "Formatted at 2026-06-20 18:39:10 on dist-test-slave-wp6g"
server_key: "deb2cda82e7f16a29762e4aac7ecc921"
server_key_iv: "cd4407109211e5af2a8754f3d44535d5"
server_key_version: "encryptionkey@0"
I20260620 18:39:10.901768 13325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "44c5a2ce2b1148389b55f7570c243f01"
format_stamp: "Formatted at 2026-06-20 18:39:10 on dist-test-slave-wp6g"
server_key: "deb2cda82e7f16a29762e4aac7ecc921"
server_key_iv: "cd4407109211e5af2a8754f3d44535d5"
server_key_version: "encryptionkey@0"
I20260620 18:39:10.903246 13325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:10.903982 13340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:10.904151 13325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:10.904255 13325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "44c5a2ce2b1148389b55f7570c243f01"
format_stamp: "Formatted at 2026-06-20 18:39:10 on dist-test-slave-wp6g"
server_key: "deb2cda82e7f16a29762e4aac7ecc921"
server_key_iv: "cd4407109211e5af2a8754f3d44535d5"
server_key_version: "encryptionkey@0"
I20260620 18:39:10.904327 13325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:10.927021 13325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:10.929461 13325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:10.929651 13325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:10.930228 13325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:10.930291 13325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:10.930336 13325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:10.930364 13325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:39:10 dist-test-slave-wp6g krb5kdc[12928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980750, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:10.937942 13325 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:36527
I20260620 18:39:10.938299 13325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:10.938323 13455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:36527 every 8 connection(s)
I20260620 18:39:10.941061 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.930825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:51147 (local address 127.10.250.126:42093)
0620 18:39:10.930899 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:10.930901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:10.931205 (+ 304us) server_negotiation.cc:407] Connection header received
0620 18:39:10.931939 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:10.931940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:10.931951 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:10.931999 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:10.932995 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.933882 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.935120 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.935325 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.935414 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.937724 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:10.937726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:10.937726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:10.937776 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:10.939570 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:10.940051 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:10.940053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:10.940053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:10.940120 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:10.940387 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:10.940388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:10.940388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:10.940517 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:10.940588 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:10.940875 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 18:39:10.940968 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:39:10.941249 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.930821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:42093 (local address 127.10.250.67:51147)
0620 18:39:10.931027 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:10.931079 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:10.931822 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:10.932084 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:10.932089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:10.932525 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:10.932855 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.932863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.933983 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.933984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.935007 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.935010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.935436 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.935437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.935574 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.936121 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:10.936130 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:10.937616 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:10.939698 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:10.939704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:10.939707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:10.939943 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:10.940191 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:10.940192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:10.940193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:10.940309 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:10.940592 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:10.940599 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:10.940739 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 18:39:10.940852 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:39:10.941018 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
I20260620 18:39:10.941744 13456 heartbeater.cc:347] Connected to a master server at 127.10.250.126:42093
I20260620 18:39:10.941824 13456 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:10.942066 13456 heartbeater.cc:511] Master 127.10.250.126:42093 requested a full tablet report, sending...
I20260620 18:39:10.942505 12973 ts_manager.cc:194] Registered new tserver with Master: 44c5a2ce2b1148389b55f7570c243f01 (127.10.250.67:36527)
I20260620 18:39:10.943056 12973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:51147
I20260620 18:39:10.948031 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 13325
I20260620 18:39:10.948094 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:10.948221 11241 external_mini_cluster.cc:1428] Setting key f498e78204553c88bd48ce80edc6e30b
I20260620 18:39:10.949175 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:39:10.949348 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.942824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:42093 (local address 127.10.250.67:40897)
0620 18:39:10.942959 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:10.942970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:10.943020 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:10.943340 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:10.943341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:10.943571 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:10.943632 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.943633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.944341 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.944342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.944997 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.944999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.945319 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.945320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.945373 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.945833 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:10.945838 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:10.946156 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:10.947705 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:10.947706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:10.947707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:10.947871 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:10.948507 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:10.948508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:10.948508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:10.948583 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:10.948991 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:10.948992 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:10.949038 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 18:39:10.949141 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:39:10.949244 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260620 18:39:10.949342 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.942922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:40897 (local address 127.10.250.126:42093)
0620 18:39:10.943213 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:10.943214 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:10.943222 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:39:10.943240 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:10.943241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:10.943251 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:10.943305 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:10.943715 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.944238 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.945074 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.945254 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.945308 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.946250 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:10.946252 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:10.946252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:10.946258 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:10.947619 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:10.948333 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:10.948334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:10.948334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:10.948381 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:10.948768 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:10.948784 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:10.948784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:10.948893 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:10.948997 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:10.949168 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 18:39:10.949242 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270}
Jun 20 18:39:10 dist-test-slave-wp6g krb5kdc[12928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:39:10 dist-test-slave-wp6g krb5kdc[12928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:10.971688 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.964530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.10.250.126:42093)
0620 18:39:10.964594 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:10.964595 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:10.964604 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:10.964629 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:10.964630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:10.964640 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:10.964701 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:10.965309 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.965807 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.966581 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.966773 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.966827 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.968935 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:10.968936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:10.968936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:10.968943 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:10.970330 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:10.970848 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:10.970850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:10.970850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:10.970906 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:10.971162 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:10.971163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:10.971164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:10.971319 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:10.971401 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:10.971554 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 18:39:10.971618 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 18:39:10.973462 12973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:10.975212 12973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:10.981586 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.977455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:42789 (local address 127.0.0.1:45586)
0620 18:39:10.977568 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:10.977610 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:10.977670 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:10.978036 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:10.978039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:10.978045 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:10.978124 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.978126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.979265 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.979266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.980448 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.980451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.981275 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.981276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.981352 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.981366 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 18:39:10.981439 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 18:39:10.981494 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":68,"threads_started":1}
I20260620 18:39:10.981597 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.977562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.10.250.65:42789)
0620 18:39:10.977735 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:10.977743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:10.977775 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:39:10.977821 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:10.977824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:10.977844 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:10.977931 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:10.978225 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.979178 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.980550 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.981190 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.981278 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.981342 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:10.981460 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 18:39:10.981500 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260620 18:39:10.981840 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.977856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:36527 (local address 127.0.0.1:45928)
0620 18:39:10.978059 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:10.978069 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:10.978121 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:10.978354 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:10.978355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:10.978357 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:10.978399 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.978401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.979709 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.979710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.980776 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.980779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.981649 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.981650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.981704 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.981707 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:10.981734 (+ 27us) client_negotiation.cc:240] Negotiation successful
0620 18:39:10.981769 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":9,"thread_start_us":65,"threads_started":1}
I20260620 18:39:10.981969 13475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.977669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:39027 (local address 127.0.0.1:44156)
0620 18:39:10.977809 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:10.977820 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:10.977876 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:10.978483 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:10.978484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:10.978486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:10.978552 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.978554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.979764 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.979765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.980582 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.980584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.981554 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.981555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.981622 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.981627 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:39:10.981673 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 18:39:10.981910 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260620 18:39:10.982184 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.977956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.10.250.67:36527)
0620 18:39:10.978162 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:10.978167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:10.978177 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:39:10.978220 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:10.978222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:10.978237 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:10.978319 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:10.978480 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.979618 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.980863 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.981550 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.981947 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.982015 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:10.982044 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 18:39:10.982094 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
I20260620 18:39:10.982708 13110 tablet_service.cc:1467] Processing CreateTablet for tablet 8504a63ed6b246f896825091f7a0f4be (DEFAULT_TABLE table=test-table [id=4ad19bb52ecb407cb946b1c5ded335eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:10.982869 13390 tablet_service.cc:1467] Processing CreateTablet for tablet 8504a63ed6b246f896825091f7a0f4be (DEFAULT_TABLE table=test-table [id=4ad19bb52ecb407cb946b1c5ded335eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:10.983107 13110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8504a63ed6b246f896825091f7a0f4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:10.983244 13390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8504a63ed6b246f896825091f7a0f4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:10.985724 13479 tablet_bootstrap.cc:492] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01: Bootstrap starting.
I20260620 18:39:10.985947 13480 tablet_bootstrap.cc:492] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072: Bootstrap starting.
I20260620 18:39:10.986182 13478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.978067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.10.250.66:39027)
0620 18:39:10.978265 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:10.978269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:10.978278 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:10.978314 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:10.978316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:10.978328 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:10.978395 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:10.978643 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.979698 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.980682 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.981394 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.981465 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.981518 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:10.982131 (+ 613us) server_negotiation.cc:299] Negotiation successful
0620 18:39:10.986088 (+ 3957us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":118,"threads_started":1}
I20260620 18:39:10.986670 13480 tablet_bootstrap.cc:654] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:10.986819 13250 tablet_service.cc:1467] Processing CreateTablet for tablet 8504a63ed6b246f896825091f7a0f4be (DEFAULT_TABLE table=test-table [id=4ad19bb52ecb407cb946b1c5ded335eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:10.987000 13480 log.cc:824] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:10.987056 13479 tablet_bootstrap.cc:654] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:10.987154 13250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8504a63ed6b246f896825091f7a0f4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:10.987442 13479 log.cc:824] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:10.987816 13480 tablet_bootstrap.cc:492] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072: No bootstrap required, opened a new log
I20260620 18:39:10.987901 13480 ts_tablet_manager.cc:1397] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:39:10.988276 13479 tablet_bootstrap.cc:492] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01: No bootstrap required, opened a new log
I20260620 18:39:10.988348 13479 ts_tablet_manager.cc:1397] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 18:39:10.989956 13480 raft_consensus.cc:348] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:10.990140 13480 raft_consensus.cc:374] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:10.990180 13480 raft_consensus.cc:729] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41c569c07254088a3c061f18a978072, State: Initialized, Role: FOLLOWER
I20260620 18:39:10.990324 13480 consensus_queue.cc:260] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:10.990275 13479 raft_consensus.cc:348] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:10.990414 13479 raft_consensus.cc:374] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:10.990422 13483 tablet_bootstrap.cc:492] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780: Bootstrap starting.
I20260620 18:39:10.990451 13479 raft_consensus.cc:729] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c5a2ce2b1148389b55f7570c243f01, State: Initialized, Role: FOLLOWER
I20260620 18:39:10.990624 13480 ts_tablet_manager.cc:1428] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 18:39:10.990607 13479 consensus_queue.cc:260] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:10.990758 13176 heartbeater.cc:503] Master 127.10.250.126:42093 was elected leader, sending a full tablet report...
I20260620 18:39:10.990866 13479 ts_tablet_manager.cc:1428] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 18:39:10.990934 13456 heartbeater.cc:503] Master 127.10.250.126:42093 was elected leader, sending a full tablet report...
I20260620 18:39:10.991447 13483 tablet_bootstrap.cc:654] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:10.991766 13483 log.cc:824] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:10.992513 13483 tablet_bootstrap.cc:492] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780: No bootstrap required, opened a new log
I20260620 18:39:10.992566 13483 ts_tablet_manager.cc:1397] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:39:10.994076 13483 raft_consensus.cc:348] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:10.994206 13483 raft_consensus.cc:374] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:10.994228 13483 raft_consensus.cc:729] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2794a8e11097434d99c6cf6fe061f780, State: Initialized, Role: FOLLOWER
I20260620 18:39:10.994303 13483 consensus_queue.cc:260] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:10.994488 13483 ts_tablet_manager.cc:1428] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:39:10.994593 13316 heartbeater.cc:503] Master 127.10.250.126:42093 was elected leader, sending a full tablet report...
W20260620 18:39:11.019205 13317 tablet.cc:2369] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:39:11.112287 13177 tablet.cc:2369] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:39:11.188951 13457 tablet.cc:2369] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:39:11.260217 13484 raft_consensus.cc:482] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:11.260344 13484 raft_consensus.cc:504] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.260766 13484 leader_election.cc:288] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2794a8e11097434d99c6cf6fe061f780 (127.10.250.66:39027), 44c5a2ce2b1148389b55f7570c243f01 (127.10.250.67:36527)
I20260620 18:39:11.264155 13478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.261148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:37585 (local address 127.10.250.66:39027)
0620 18:39:11.261216 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:11.261218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:11.261233 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:39:11.261321 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:11.261323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:11.261333 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:11.261411 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:11.261597 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.262156 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.263287 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.263772 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.263857 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.263869 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:11.264024 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 18:39:11.264071 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:39:11.264178 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.261407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:33687 (local address 127.10.250.67:36527)
0620 18:39:11.261457 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:11.261459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:11.261466 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:39:11.261490 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:11.261491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:11.261500 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:11.261559 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:11.261796 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.262343 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.263416 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.263774 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.263891 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.263900 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:11.264076 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 18:39:11.264110 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 18:39:11.264165 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.261050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:39027 (local address 127.10.250.65:37585)
0620 18:39:11.261180 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:11.261191 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:11.261245 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:11.261430 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:11.261431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:11.261434 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:11.261525 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.261529 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.262262 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.262264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.263208 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.263211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.263867 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.263868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.263942 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.263947 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:39:11.264002 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 18:39:11.264066 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260620 18:39:11.264178 13489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.261224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:36527 (local address 127.10.250.65:33687)
0620 18:39:11.261352 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:11.261363 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:11.261414 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:11.261594 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:11.261595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:11.261597 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:11.261682 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.261683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.262435 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.262436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.263327 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.263330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.263902 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.263904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.263975 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.263978 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:11.264054 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 18:39:11.264093 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":54,"threads_started":1}
I20260620 18:39:11.264619 13270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8504a63ed6b246f896825091f7a0f4be" candidate_uuid: "a41c569c07254088a3c061f18a978072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794a8e11097434d99c6cf6fe061f780" is_pre_election: true
I20260620 18:39:11.264820 13270 raft_consensus.cc:2454] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41c569c07254088a3c061f18a978072 in term 0.
I20260620 18:39:11.264822 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8504a63ed6b246f896825091f7a0f4be" candidate_uuid: "a41c569c07254088a3c061f18a978072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c5a2ce2b1148389b55f7570c243f01" is_pre_election: true
I20260620 18:39:11.264998 13410 raft_consensus.cc:2454] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41c569c07254088a3c061f18a978072 in term 0.
I20260620 18:39:11.265081 13063 leader_election.cc:302] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2794a8e11097434d99c6cf6fe061f780, a41c569c07254088a3c061f18a978072; no voters:
I20260620 18:39:11.265254 13484 raft_consensus.cc:2781] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:11.265354 13484 raft_consensus.cc:482] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:11.265388 13484 raft_consensus.cc:3037] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:11.266127 13484 raft_consensus.cc:504] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.266286 13484 leader_election.cc:288] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [CANDIDATE]: Term 1 election: Requested vote from peers 2794a8e11097434d99c6cf6fe061f780 (127.10.250.66:39027), 44c5a2ce2b1148389b55f7570c243f01 (127.10.250.67:36527)
I20260620 18:39:11.266479 13270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8504a63ed6b246f896825091f7a0f4be" candidate_uuid: "a41c569c07254088a3c061f18a978072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794a8e11097434d99c6cf6fe061f780"
I20260620 18:39:11.266548 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8504a63ed6b246f896825091f7a0f4be" candidate_uuid: "a41c569c07254088a3c061f18a978072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c5a2ce2b1148389b55f7570c243f01"
I20260620 18:39:11.266594 13270 raft_consensus.cc:3037] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:11.266635 13410 raft_consensus.cc:3037] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:11.267328 13270 raft_consensus.cc:2454] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41c569c07254088a3c061f18a978072 in term 1.
I20260620 18:39:11.267558 13063 leader_election.cc:302] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2794a8e11097434d99c6cf6fe061f780, a41c569c07254088a3c061f18a978072; no voters:
I20260620 18:39:11.267659 13484 raft_consensus.cc:2781] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:11.267652 13410 raft_consensus.cc:2454] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41c569c07254088a3c061f18a978072 in term 1.
I20260620 18:39:11.267781 13484 raft_consensus.cc:686] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [term 1 LEADER]: Becoming Leader. State: Replica: a41c569c07254088a3c061f18a978072, State: Running, Role: LEADER
I20260620 18:39:11.267886 13484 consensus_queue.cc:237] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.268786 12973 catalog_manager.cc:5495] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 reported cstate change: term changed from 0 to 1, leader changed from <none> to a41c569c07254088a3c061f18a978072 (127.10.250.65). New cstate: current_term: 1 leader_uuid: "a41c569c07254088a3c061f18a978072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } health_report { overall_health: UNKNOWN } } }
Jun 20 18:39:11 dist-test-slave-wp6g krb5kdc[12928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980750, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:11.361917 13497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.355303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:42093 (local address 127.0.0.1:37374)
0620 18:39:11.355536 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:11.355548 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:11.355619 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:11.355780 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:11.355781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:11.356092 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:11.356168 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.356170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.356802 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.356802 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.357340 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.357341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.357645 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.357646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.357683 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.358195 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:11.358199 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:11.359550 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:11.360956 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:11.360962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:11.360964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:11.361180 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:11.361357 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:11.361358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:11.361358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:11.361414 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:11.361618 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:11.361636 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:11.361678 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 18:39:11.361730 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 18:39:11.361823 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":127,"threads_started":1}
I20260620 18:39:11.361967 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.355347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.10.250.126:42093)
0620 18:39:11.355421 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:11.355424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:11.355659 (+ 235us) server_negotiation.cc:407] Connection header received
0620 18:39:11.355689 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:11.355690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:11.355701 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:11.355759 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:11.356240 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.356733 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.357409 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.357583 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.357632 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.359626 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:11.359627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:11.359627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:11.359634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:11.360877 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:11.361247 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:11.361247 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:11.361248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:11.361299 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:11.361465 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:11.361466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:11.361466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:11.361566 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:11.361614 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:11.361761 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 18:39:11.361870 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":43}
I20260620 18:39:11.364174 12973 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:37374:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:39:11.364374 12973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:11.366010 13250 tablet_service.cc:1467] Processing CreateTablet for tablet cb80240436d8409289442d697de168ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c90a0fb867ae49c4bed31fb70f74d66d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:39:11.366020 13110 tablet_service.cc:1467] Processing CreateTablet for tablet cb80240436d8409289442d697de168ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c90a0fb867ae49c4bed31fb70f74d66d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:39:11.366201 13110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb80240436d8409289442d697de168ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:11.366201 13250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb80240436d8409289442d697de168ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:11.367574 13480 tablet_bootstrap.cc:492] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072: Bootstrap starting.
I20260620 18:39:11.367655 13483 tablet_bootstrap.cc:492] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780: Bootstrap starting.
I20260620 18:39:11.367710 13390 tablet_service.cc:1467] Processing CreateTablet for tablet cb80240436d8409289442d697de168ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c90a0fb867ae49c4bed31fb70f74d66d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:39:11.367841 13390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb80240436d8409289442d697de168ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:11.368033 13483 tablet_bootstrap.cc:654] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:11.368033 13480 tablet_bootstrap.cc:654] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:11.368710 13480 tablet_bootstrap.cc:492] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072: No bootstrap required, opened a new log
I20260620 18:39:11.368779 13480 ts_tablet_manager.cc:1397] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:11.368929 13483 tablet_bootstrap.cc:492] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780: No bootstrap required, opened a new log
I20260620 18:39:11.368971 13483 ts_tablet_manager.cc:1397] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:11.369060 13479 tablet_bootstrap.cc:492] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01: Bootstrap starting.
I20260620 18:39:11.369127 13483 raft_consensus.cc:348] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.369156 13480 raft_consensus.cc:348] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.369208 13483 raft_consensus.cc:374] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:11.369220 13480 raft_consensus.cc:374] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:11.369231 13483 raft_consensus.cc:729] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2794a8e11097434d99c6cf6fe061f780, State: Initialized, Role: FOLLOWER
I20260620 18:39:11.369244 13480 raft_consensus.cc:729] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41c569c07254088a3c061f18a978072, State: Initialized, Role: FOLLOWER
I20260620 18:39:11.369292 13480 consensus_queue.cc:260] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.369283 13483 consensus_queue.cc:260] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.369376 13480 ts_tablet_manager.cc:1428] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:11.369391 13483 ts_tablet_manager.cc:1428] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:11.369505 13484 tablet_replica.cc:439] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } } }
I20260620 18:39:11.369539 13487 tablet_replica.cc:439] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } } }
I20260620 18:39:11.369586 13484 tablet_replica.cc:442] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:11.369604 13487 tablet_replica.cc:442] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:11.369627 13479 tablet_bootstrap.cc:654] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:11.370360 13479 tablet_bootstrap.cc:492] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01: No bootstrap required, opened a new log
I20260620 18:39:11.370430 13479 ts_tablet_manager.cc:1397] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 18:39:11.370604 13479 raft_consensus.cc:348] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.370690 13479 raft_consensus.cc:374] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:11.370713 13479 raft_consensus.cc:729] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c5a2ce2b1148389b55f7570c243f01, State: Initialized, Role: FOLLOWER
I20260620 18:39:11.370764 13479 consensus_queue.cc:260] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.370847 13479 ts_tablet_manager.cc:1428] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:11.370913 13485 tablet_replica.cc:439] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } } }
I20260620 18:39:11.371022 13485 tablet_replica.cc:442] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:11.425828 13484 raft_consensus.cc:482] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:11.425925 13484 raft_consensus.cc:504] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.426095 13484 leader_election.cc:288] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2794a8e11097434d99c6cf6fe061f780 (127.10.250.66:39027), 44c5a2ce2b1148389b55f7570c243f01 (127.10.250.67:36527)
I20260620 18:39:11.426318 13270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb80240436d8409289442d697de168ef" candidate_uuid: "a41c569c07254088a3c061f18a978072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794a8e11097434d99c6cf6fe061f780" is_pre_election: true
I20260620 18:39:11.426400 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb80240436d8409289442d697de168ef" candidate_uuid: "a41c569c07254088a3c061f18a978072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c5a2ce2b1148389b55f7570c243f01" is_pre_election: true
I20260620 18:39:11.426456 13270 raft_consensus.cc:2454] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41c569c07254088a3c061f18a978072 in term 0.
I20260620 18:39:11.426484 13410 raft_consensus.cc:2454] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41c569c07254088a3c061f18a978072 in term 0.
I20260620 18:39:11.426648 13063 leader_election.cc:302] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c5a2ce2b1148389b55f7570c243f01, a41c569c07254088a3c061f18a978072; no voters:
I20260620 18:39:11.426757 13484 raft_consensus.cc:2781] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:11.426813 13484 raft_consensus.cc:482] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:11.426841 13484 raft_consensus.cc:3037] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:11.427567 13484 raft_consensus.cc:504] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.427724 13484 leader_election.cc:288] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [CANDIDATE]: Term 1 election: Requested vote from peers 2794a8e11097434d99c6cf6fe061f780 (127.10.250.66:39027), 44c5a2ce2b1148389b55f7570c243f01 (127.10.250.67:36527)
I20260620 18:39:11.427958 13270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb80240436d8409289442d697de168ef" candidate_uuid: "a41c569c07254088a3c061f18a978072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794a8e11097434d99c6cf6fe061f780"
I20260620 18:39:11.428058 13270 raft_consensus.cc:3037] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:11.428069 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb80240436d8409289442d697de168ef" candidate_uuid: "a41c569c07254088a3c061f18a978072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c5a2ce2b1148389b55f7570c243f01"
I20260620 18:39:11.428200 13410 raft_consensus.cc:3037] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:11.428750 13270 raft_consensus.cc:2454] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41c569c07254088a3c061f18a978072 in term 1.
I20260620 18:39:11.428902 13410 raft_consensus.cc:2454] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41c569c07254088a3c061f18a978072 in term 1.
I20260620 18:39:11.428949 13063 leader_election.cc:302] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2794a8e11097434d99c6cf6fe061f780, a41c569c07254088a3c061f18a978072; no voters:
I20260620 18:39:11.429041 13484 raft_consensus.cc:2781] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:11.429091 13484 raft_consensus.cc:686] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [term 1 LEADER]: Becoming Leader. State: Replica: a41c569c07254088a3c061f18a978072, State: Running, Role: LEADER
I20260620 18:39:11.429138 13484 consensus_queue.cc:237] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.429512 13490 tablet_replica.cc:439] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072: TxnStatusTablet state changed. Reason: New leader a41c569c07254088a3c061f18a978072. Latest consensus state: current_term: 1 leader_uuid: "a41c569c07254088a3c061f18a978072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } } }
I20260620 18:39:11.429606 13490 tablet_replica.cc:442] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:39:11.429818 13502 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:39:11.429818 12973 catalog_manager.cc:5495] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 reported cstate change: term changed from 0 to 1, leader changed from <none> to a41c569c07254088a3c061f18a978072 (127.10.250.65). New cstate: current_term: 1 leader_uuid: "a41c569c07254088a3c061f18a978072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } health_report { overall_health: UNKNOWN } } }
I20260620 18:39:11.429967 13502 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:39:11.445621 13497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.442635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:42789 (local address 127.0.0.1:45592)
0620 18:39:11.442698 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:11.442712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:11.442787 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:11.442966 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:11.442966 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:11.442968 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:39:11.443057 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.443060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.443978 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.443979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.444629 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.444632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.445103 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.445104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.445178 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.445193 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:39:11.445488 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:39:11.445495 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:39:11.445523 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 18:39:11.445561 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 18:39:11.445636 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.442696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.10.250.65:42789)
0620 18:39:11.442747 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:11.442750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:11.442760 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:39:11.442868 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:11.442869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:11.442890 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:11.442965 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:11.443132 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.443902 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.444709 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.445025 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.445097 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.445289 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:11.445423 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:11.445504 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:11.445529 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 18:39:11.445554 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:39:11.447626 13410 raft_consensus.cc:1261] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01 [term 1 FOLLOWER]: Refusing update from remote peer a41c569c07254088a3c061f18a978072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:11.447664 13270 raft_consensus.cc:1261] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Refusing update from remote peer a41c569c07254088a3c061f18a978072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:11.447976 13484 consensus_queue.cc:1035] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:11.448186 13484 consensus_queue.cc:1035] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:11.449189 13485 tablet_replica.cc:439] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01: TxnStatusTablet state changed. Reason: New leader a41c569c07254088a3c061f18a978072. Latest consensus state: current_term: 1 leader_uuid: "a41c569c07254088a3c061f18a978072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } } }
I20260620 18:39:11.449290 13485 tablet_replica.cc:442] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:11.449451 13487 tablet_replica.cc:439] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780: TxnStatusTablet state changed. Reason: New leader a41c569c07254088a3c061f18a978072. Latest consensus state: current_term: 1 leader_uuid: "a41c569c07254088a3c061f18a978072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } } }
I20260620 18:39:11.449589 13487 tablet_replica.cc:442] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:11.451197 13484 tablet_replica.cc:439] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a41c569c07254088a3c061f18a978072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } } }
I20260620 18:39:11.451279 13484 tablet_replica.cc:442] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:39:11.451399 13487 tablet_replica.cc:439] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a41c569c07254088a3c061f18a978072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } } }
I20260620 18:39:11.451483 13487 tablet_replica.cc:442] T cb80240436d8409289442d697de168ef P 2794a8e11097434d99c6cf6fe061f780: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:11.451716 13507 mvcc.cc:205] Tried to move back new op lower bound from 7298993157926481920 to 7298993157854253056. Current Snapshot: MvccSnapshot[applied={T|T < 7298993157926481920}]
I20260620 18:39:11.452196 13505 mvcc.cc:205] Tried to move back new op lower bound from 7298993157926481920 to 7298993157854253056. Current Snapshot: MvccSnapshot[applied={T|T < 7298993157926481920 or (T in {7298993157926481920})}]
I20260620 18:39:11.455752 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.453144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45606 (local address 127.10.250.65:42789)
0620 18:39:11.453184 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:11.453186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:11.453196 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:39:11.453247 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:11.453247 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:11.453257 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:11.453322 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:11.453416 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.454187 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.455021 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.455349 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.455424 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.455528 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:11.455564 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:11.455612 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:11.455660 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 18:39:11.455680 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:39:11.461469 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.458608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:47291 (local address 127.10.250.65:42789)
0620 18:39:11.458669 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:11.458671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:11.458785 (+ 114us) server_negotiation.cc:407] Connection header received
0620 18:39:11.458822 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:11.458823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:11.458834 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:11.458892 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:11.459041 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.459750 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.460534 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.460828 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.460900 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.461211 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:11.461242 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:11.461296 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:11.461359 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 18:39:11.461378 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:39:11.461627 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.458531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:42789 (local address 127.10.250.65:47291)
0620 18:39:11.458710 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:11.458719 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:11.458782 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:11.458909 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:11.458910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:11.458911 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:39:11.458967 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.458968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.459828 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.459828 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.460447 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.460449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.460983 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.460985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.461046 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.461062 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:39:11.461309 (+ 247us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:39:11.461310 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 18:39:11.461342 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 18:39:11.461383 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260620 18:39:11.463615 13516 tablet_replica.cc:439] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a41c569c07254088a3c061f18a978072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } } }
I20260620 18:39:11.463734 13516 tablet_replica.cc:442] T cb80240436d8409289442d697de168ef P a41c569c07254088a3c061f18a978072: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:39:11.464062 13506 mvcc.cc:205] Tried to move back new op lower bound from 7298993157926481920 to 7298993157854253056. Current Snapshot: MvccSnapshot[applied={T|T < 7298993157926481920}]
I20260620 18:39:11.464149 13409 raft_consensus.cc:1261] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01 [term 1 FOLLOWER]: Refusing update from remote peer a41c569c07254088a3c061f18a978072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:11.464283 13485 tablet_replica.cc:439] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a41c569c07254088a3c061f18a978072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } } }
I20260620 18:39:11.464362 13485 tablet_replica.cc:442] T cb80240436d8409289442d697de168ef P 44c5a2ce2b1148389b55f7570c243f01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:11.464533 13270 raft_consensus.cc:1261] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Refusing update from remote peer a41c569c07254088a3c061f18a978072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:11.464368 13516 consensus_queue.cc:1035] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:11.464936 13490 consensus_queue.cc:1035] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:11.479176 12973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:11.479302 12973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:11.480119 12973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f: Sending DeleteTablet for 3 replicas of tablet 8504a63ed6b246f896825091f7a0f4be
I20260620 18:39:11.480494 13109 tablet_service.cc:1514] Processing DeleteTablet for tablet 8504a63ed6b246f896825091f7a0f4be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:11 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:45586
I20260620 18:39:11.480470 13250 tablet_service.cc:1514] Processing DeleteTablet for tablet 8504a63ed6b246f896825091f7a0f4be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:11 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:44156
I20260620 18:39:11.480545 13390 tablet_service.cc:1514] Processing DeleteTablet for tablet 8504a63ed6b246f896825091f7a0f4be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:11 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:45928
I20260620 18:39:11.480983 13523 tablet_replica.cc:330] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780: stopping tablet replica
I20260620 18:39:11.481035 13524 tablet_replica.cc:330] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01: stopping tablet replica
I20260620 18:39:11.481001 12973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:39:11.481112 13523 raft_consensus.cc:2229] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:11.481127 13524 raft_consensus.cc:2229] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 18:39:11.481153 12973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:11.481170 13523 raft_consensus.cc:2258] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:11.481175 13524 raft_consensus.cc:2258] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:11.481673 13523 ts_tablet_manager.cc:1905] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:11.481336 13522 tablet_replica.cc:330] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072: stopping tablet replica
I20260620 18:39:11.482116 13522 raft_consensus.cc:2229] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:11.482232 13522 raft_consensus.cc:2258] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:11.482544 13250 tablet_service.cc:1467] Processing CreateTablet for tablet 6d84bbc30b0f4c288afd0c69a27e5c94 (DEFAULT_TABLE table=test-table [id=f2e821575b12489d9685f1b795749545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:11.482589 13109 tablet_service.cc:1467] Processing CreateTablet for tablet 6d84bbc30b0f4c288afd0c69a27e5c94 (DEFAULT_TABLE table=test-table [id=f2e821575b12489d9685f1b795749545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:11.482625 13390 tablet_service.cc:1467] Processing CreateTablet for tablet 6d84bbc30b0f4c288afd0c69a27e5c94 (DEFAULT_TABLE table=test-table [id=f2e821575b12489d9685f1b795749545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:11.482662 13250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d84bbc30b0f4c288afd0c69a27e5c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:11.482682 13109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d84bbc30b0f4c288afd0c69a27e5c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:11.482751 13390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d84bbc30b0f4c288afd0c69a27e5c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:11.483985 13483 tablet_bootstrap.cc:492] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780: Bootstrap starting.
I20260620 18:39:11.484014 13479 tablet_bootstrap.cc:492] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01: Bootstrap starting.
I20260620 18:39:11.484246 13480 tablet_bootstrap.cc:492] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072: Bootstrap starting.
I20260620 18:39:11.484411 13483 tablet_bootstrap.cc:654] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:11.484468 13479 tablet_bootstrap.cc:654] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:11.484655 13480 tablet_bootstrap.cc:654] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:11.485240 13522 ts_tablet_manager.cc:1905] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:11.485261 13480 tablet_bootstrap.cc:492] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072: No bootstrap required, opened a new log
I20260620 18:39:11.485311 13480 ts_tablet_manager.cc:1397] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:11.485471 13480 raft_consensus.cc:348] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.485563 13480 raft_consensus.cc:374] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:11.485584 13480 raft_consensus.cc:729] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41c569c07254088a3c061f18a978072, State: Initialized, Role: FOLLOWER
I20260620 18:39:11.485627 13480 consensus_queue.cc:260] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.485735 13480 ts_tablet_manager.cc:1428] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:11.486730 13523 ts_tablet_manager.cc:1918] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:11.486816 13523 log.cc:1196] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/8504a63ed6b246f896825091f7a0f4be
I20260620 18:39:11.486850 13522 ts_tablet_manager.cc:1918] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:11.486917 13522 log.cc:1196] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/wal/wals/8504a63ed6b246f896825091f7a0f4be
I20260620 18:39:11.487109 13523 ts_tablet_manager.cc:1939] T 8504a63ed6b246f896825091f7a0f4be P 2794a8e11097434d99c6cf6fe061f780: Deleting consensus metadata
I20260620 18:39:11.487161 13522 ts_tablet_manager.cc:1939] T 8504a63ed6b246f896825091f7a0f4be P a41c569c07254088a3c061f18a978072: Deleting consensus metadata
I20260620 18:39:11.487520 12959 catalog_manager.cc:4841] TS a41c569c07254088a3c061f18a978072 (127.10.250.65:42789): tablet 8504a63ed6b246f896825091f7a0f4be (table test-table [id=4ad19bb52ecb407cb946b1c5ded335eb]) successfully deleted
I20260620 18:39:11.487622 12959 catalog_manager.cc:4841] TS 2794a8e11097434d99c6cf6fe061f780 (127.10.250.66:39027): tablet 8504a63ed6b246f896825091f7a0f4be (table test-table [id=4ad19bb52ecb407cb946b1c5ded335eb]) successfully deleted
I20260620 18:39:11.487768 13483 tablet_bootstrap.cc:492] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780: No bootstrap required, opened a new log
I20260620 18:39:11.487816 13483 ts_tablet_manager.cc:1397] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 18:39:11.487957 13483 raft_consensus.cc:348] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.488049 13483 raft_consensus.cc:374] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:11.488080 13483 raft_consensus.cc:729] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2794a8e11097434d99c6cf6fe061f780, State: Initialized, Role: FOLLOWER
I20260620 18:39:11.488142 13483 consensus_queue.cc:260] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.488235 13483 ts_tablet_manager.cc:1428] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:11.488273 13479 tablet_bootstrap.cc:492] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01: No bootstrap required, opened a new log
I20260620 18:39:11.488320 13479 ts_tablet_manager.cc:1397] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260620 18:39:11.488461 13479 raft_consensus.cc:348] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.488535 13479 raft_consensus.cc:374] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:11.488554 13479 raft_consensus.cc:729] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c5a2ce2b1148389b55f7570c243f01, State: Initialized, Role: FOLLOWER
I20260620 18:39:11.488601 13479 consensus_queue.cc:260] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.488682 13479 ts_tablet_manager.cc:1428] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:11.488927 13524 ts_tablet_manager.cc:1905] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:11.490586 13524 ts_tablet_manager.cc:1918] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:11.490659 13524 log.cc:1196] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/8504a63ed6b246f896825091f7a0f4be
I20260620 18:39:11.490922 13524 ts_tablet_manager.cc:1939] T 8504a63ed6b246f896825091f7a0f4be P 44c5a2ce2b1148389b55f7570c243f01: Deleting consensus metadata
I20260620 18:39:11.491356 12959 catalog_manager.cc:4841] TS 44c5a2ce2b1148389b55f7570c243f01 (127.10.250.67:36527): tablet 8504a63ed6b246f896825091f7a0f4be (table test-table [id=4ad19bb52ecb407cb946b1c5ded335eb]) successfully deleted
I20260620 18:39:11.573109 13485 raft_consensus.cc:482] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:11.573237 13485 raft_consensus.cc:504] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.573601 13485 leader_election.cc:288] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41c569c07254088a3c061f18a978072 (127.10.250.65:42789), 2794a8e11097434d99c6cf6fe061f780 (127.10.250.66:39027)
I20260620 18:39:11.577538 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.574030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:39027 (local address 127.10.250.67:60599)
0620 18:39:11.574118 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:11.574130 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:11.574220 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:11.574395 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:11.574396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:11.574398 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:11.574464 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.574466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.575451 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.575453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.576474 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.576477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.577264 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.577265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.577336 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.577341 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:39:11.577399 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 18:39:11.577439 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":24,"thread_start_us":38,"threads_started":1}
I20260620 18:39:11.577544 13478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.574093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:60599 (local address 127.10.250.66:39027)
0620 18:39:11.574137 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:11.574139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:11.574192 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:39:11.574301 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:11.574303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:11.574312 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:11.574371 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:11.574563 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.575327 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.576622 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.577169 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.577252 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.577261 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:11.577420 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 18:39:11.577474 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:39:11.577610 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.573889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:56411 (local address 127.10.250.65:42789)
0620 18:39:11.573952 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:11.573954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:11.574078 (+ 124us) server_negotiation.cc:407] Connection header received
0620 18:39:11.574203 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:11.574205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:11.574216 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:11.574277 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:11.574610 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.575338 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.576667 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.577203 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.577262 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.577272 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:11.577505 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 18:39:11.577540 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:39:11.577934 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.573829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:42789 (local address 127.10.250.67:56411)
0620 18:39:11.573989 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:11.574002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:11.574108 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:11.574417 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:11.574417 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:11.574419 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:11.574492 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.574494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.575414 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.575415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.576596 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.576599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.577326 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.577327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.577419 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.577424 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:39:11.577708 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 18:39:11.577751 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":31,"thread_start_us":86,"threads_started":1}
I20260620 18:39:11.577975 13270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d84bbc30b0f4c288afd0c69a27e5c94" candidate_uuid: "44c5a2ce2b1148389b55f7570c243f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794a8e11097434d99c6cf6fe061f780" is_pre_election: true
I20260620 18:39:11.578081 13270 raft_consensus.cc:2454] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c5a2ce2b1148389b55f7570c243f01 in term 0.
I20260620 18:39:11.578150 13130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d84bbc30b0f4c288afd0c69a27e5c94" candidate_uuid: "44c5a2ce2b1148389b55f7570c243f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41c569c07254088a3c061f18a978072" is_pre_election: true
I20260620 18:39:11.578285 13130 raft_consensus.cc:2454] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c5a2ce2b1148389b55f7570c243f01 in term 0.
I20260620 18:39:11.578284 13343 leader_election.cc:302] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2794a8e11097434d99c6cf6fe061f780, 44c5a2ce2b1148389b55f7570c243f01; no voters:
I20260620 18:39:11.578428 13485 raft_consensus.cc:2781] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:11.578464 13485 raft_consensus.cc:482] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:11.578495 13485 raft_consensus.cc:3037] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:11.579241 13485 raft_consensus.cc:504] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.579388 13485 leader_election.cc:288] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [CANDIDATE]: Term 1 election: Requested vote from peers a41c569c07254088a3c061f18a978072 (127.10.250.65:42789), 2794a8e11097434d99c6cf6fe061f780 (127.10.250.66:39027)
I20260620 18:39:11.579577 13270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d84bbc30b0f4c288afd0c69a27e5c94" candidate_uuid: "44c5a2ce2b1148389b55f7570c243f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794a8e11097434d99c6cf6fe061f780"
I20260620 18:39:11.579588 13130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d84bbc30b0f4c288afd0c69a27e5c94" candidate_uuid: "44c5a2ce2b1148389b55f7570c243f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41c569c07254088a3c061f18a978072"
I20260620 18:39:11.579655 13270 raft_consensus.cc:3037] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:11.579666 13130 raft_consensus.cc:3037] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:11.580224 13270 raft_consensus.cc:2454] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c5a2ce2b1148389b55f7570c243f01 in term 1.
I20260620 18:39:11.580435 13130 raft_consensus.cc:2454] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c5a2ce2b1148389b55f7570c243f01 in term 1.
I20260620 18:39:11.580596 13343 leader_election.cc:302] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c5a2ce2b1148389b55f7570c243f01, a41c569c07254088a3c061f18a978072; no voters:
I20260620 18:39:11.580714 13485 raft_consensus.cc:2781] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:11.580886 13485 raft_consensus.cc:686] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [term 1 LEADER]: Becoming Leader. State: Replica: 44c5a2ce2b1148389b55f7570c243f01, State: Running, Role: LEADER
I20260620 18:39:11.580966 13485 consensus_queue.cc:237] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:11.581746 12973 catalog_manager.cc:5495] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44c5a2ce2b1148389b55f7570c243f01 (127.10.250.67). New cstate: current_term: 1 leader_uuid: "44c5a2ce2b1148389b55f7570c243f01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } health_report { overall_health: HEALTHY } } }
I20260620 18:39:11.605459 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.603052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.10.250.67:36527)
0620 18:39:11.603096 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:11.603098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:11.603123 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:39:11.603159 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:11.603160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:11.603181 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:11.603271 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:11.603383 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.603874 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.604647 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.604850 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.604925 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.605090 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:11.605214 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:11.605294 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:11.605346 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 18:39:11.605366 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:39:11.610627 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.608146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:45135 (local address 127.10.250.65:42789)
0620 18:39:11.608195 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:11.608197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:11.608297 (+ 100us) server_negotiation.cc:407] Connection header received
0620 18:39:11.608362 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:11.608363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:11.608375 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:11.608419 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:11.608576 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.609039 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.609845 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.610048 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.610123 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.610311 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:11.610353 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:11.610421 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:11.610486 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 18:39:11.610507 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:39:11.610659 13534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.608049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:42789 (local address 127.10.250.67:45135)
0620 18:39:11.608227 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:11.608237 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:11.608299 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:11.608429 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:11.608429 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:11.608431 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:39:11.608502 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.608505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.609127 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.609129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.609737 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.609739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.610124 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.610126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.610198 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.610223 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:39:11.610424 (+ 201us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:39:11.610427 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:11.610481 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 18:39:11.610551 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260620 18:39:11.615993 13270 raft_consensus.cc:1261] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Refusing update from remote peer 44c5a2ce2b1148389b55f7570c243f01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:11.616319 13485 consensus_queue.cc:1035] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:11.616439 13130 raft_consensus.cc:1261] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072 [term 1 FOLLOWER]: Refusing update from remote peer 44c5a2ce2b1148389b55f7570c243f01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:11.617043 13485 consensus_queue.cc:1035] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:11.624567 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.621813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:36527 (local address 127.10.250.65:55649)
0620 18:39:11.621858 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:11.621870 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:11.621930 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:11.622098 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:11.622099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:11.622102 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:39:11.622151 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.622153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.623126 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.623127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.623781 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.623784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.624232 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.624233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.624277 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.624283 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:39:11.624437 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:39:11.624438 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 18:39:11.624473 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 18:39:11.624511 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 18:39:11.624571 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.621897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:55649 (local address 127.10.250.67:36527)
0620 18:39:11.621957 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:11.621960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:11.621969 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:11.621992 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:11.621993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:11.622004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:11.622052 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:11.622226 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.623040 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.623863 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.624165 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.624234 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.624346 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:11.624380 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:11.624428 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:11.624478 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 18:39:11.624496 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 18:39:11.624946 13506 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 18:39:13.150589 12973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:13.150714 12973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:13.151480 12973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f: Sending DeleteTablet for 3 replicas of tablet 6d84bbc30b0f4c288afd0c69a27e5c94
I20260620 18:39:13.151741 13109 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d84bbc30b0f4c288afd0c69a27e5c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:13 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:45586
I20260620 18:39:13.151815 13390 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d84bbc30b0f4c288afd0c69a27e5c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:13 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:45928
I20260620 18:39:13.151831 13250 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d84bbc30b0f4c288afd0c69a27e5c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:13 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:44156
I20260620 18:39:13.152359 13562 tablet_replica.cc:330] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072: stopping tablet replica
I20260620 18:39:13.152453 13562 raft_consensus.cc:2229] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:13.152496 13562 raft_consensus.cc:2258] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:13.153187 13562 ts_tablet_manager.cc:1905] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:13.154130 13564 tablet_replica.cc:330] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780: stopping tablet replica
I20260620 18:39:13.154218 13564 raft_consensus.cc:2229] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:13.154253 13564 raft_consensus.cc:2258] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:13.154608 13563 tablet_replica.cc:330] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01: stopping tablet replica
I20260620 18:39:13.154685 13563 raft_consensus.cc:2229] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:13.154809 13563 raft_consensus.cc:2258] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:13.155049 13563 ts_tablet_manager.cc:1905] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:13.155072 13562 ts_tablet_manager.cc:1918] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:39:13.155164 13562 log.cc:1196] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/wal/wals/6d84bbc30b0f4c288afd0c69a27e5c94
I20260620 18:39:13.155395 13562 ts_tablet_manager.cc:1939] T 6d84bbc30b0f4c288afd0c69a27e5c94 P a41c569c07254088a3c061f18a978072: Deleting consensus metadata
I20260620 18:39:13.155777 12959 catalog_manager.cc:4841] TS a41c569c07254088a3c061f18a978072 (127.10.250.65:42789): tablet 6d84bbc30b0f4c288afd0c69a27e5c94 (table test-table [id=f2e821575b12489d9685f1b795749545]) successfully deleted
I20260620 18:39:13.155969 13564 ts_tablet_manager.cc:1905] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:13.156778 13563 ts_tablet_manager.cc:1918] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:39:13.156845 13563 log.cc:1196] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/6d84bbc30b0f4c288afd0c69a27e5c94
I20260620 18:39:13.157085 13563 ts_tablet_manager.cc:1939] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 44c5a2ce2b1148389b55f7570c243f01: Deleting consensus metadata
I20260620 18:39:13.157374 12959 catalog_manager.cc:4841] TS 44c5a2ce2b1148389b55f7570c243f01 (127.10.250.67:36527): tablet 6d84bbc30b0f4c288afd0c69a27e5c94 (table test-table [id=f2e821575b12489d9685f1b795749545]) successfully deleted
I20260620 18:39:13.157581 13564 ts_tablet_manager.cc:1918] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:39:13.157651 13564 log.cc:1196] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/6d84bbc30b0f4c288afd0c69a27e5c94
I20260620 18:39:13.157895 13564 ts_tablet_manager.cc:1939] T 6d84bbc30b0f4c288afd0c69a27e5c94 P 2794a8e11097434d99c6cf6fe061f780: Deleting consensus metadata
I20260620 18:39:13.158444 12959 catalog_manager.cc:4841] TS 2794a8e11097434d99c6cf6fe061f780 (127.10.250.66:39027): tablet 6d84bbc30b0f4c288afd0c69a27e5c94 (table test-table [id=f2e821575b12489d9685f1b795749545]) successfully deleted
Jun 20 18:39:13 dist-test-slave-wp6g krb5kdc[12928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:39:13 dist-test-slave-wp6g krb5kdc[12928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:13.185104 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:13.172098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37378 (local address 127.10.250.126:42093)
0620 18:39:13.172259 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:13.172262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:13.172299 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:39:13.172390 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:13.172391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:13.172404 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:13.172500 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:13.173068 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.173583 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.174368 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.174638 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.174722 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:13.182061 (+ 7339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:13.182063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:13.182063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:13.182072 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:13.183823 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:13.184214 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:13.184216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:13.184216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:13.184284 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:13.184495 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:13.184497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:13.184497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:13.184629 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:13.184741 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:13.184839 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 18:39:13.184929 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260620 18:39:13.186792 12973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:13.186957 12973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:13.188405 13109 tablet_service.cc:1467] Processing CreateTablet for tablet f6a6b1fbb77c4c6e882e1b964173f235 (DEFAULT_TABLE table=test-table [id=6c405f1d9ae94d738d26032b63e59ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:13.188426 13390 tablet_service.cc:1467] Processing CreateTablet for tablet f6a6b1fbb77c4c6e882e1b964173f235 (DEFAULT_TABLE table=test-table [id=6c405f1d9ae94d738d26032b63e59ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:13.188563 13109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a6b1fbb77c4c6e882e1b964173f235. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:13.188565 13390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a6b1fbb77c4c6e882e1b964173f235. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:13.189810 13574 tablet_bootstrap.cc:492] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072: Bootstrap starting.
I20260620 18:39:13.189813 13250 tablet_service.cc:1467] Processing CreateTablet for tablet f6a6b1fbb77c4c6e882e1b964173f235 (DEFAULT_TABLE table=test-table [id=6c405f1d9ae94d738d26032b63e59ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:13.189855 13573 tablet_bootstrap.cc:492] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01: Bootstrap starting.
I20260620 18:39:13.189934 13250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a6b1fbb77c4c6e882e1b964173f235. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:13.190222 13573 tablet_bootstrap.cc:654] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:13.190302 13574 tablet_bootstrap.cc:654] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:13.190840 13573 tablet_bootstrap.cc:492] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01: No bootstrap required, opened a new log
I20260620 18:39:13.190905 13573 ts_tablet_manager.cc:1397] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:13.191013 13573 raft_consensus.cc:348] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.191087 13573 raft_consensus.cc:374] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:13.191128 13573 raft_consensus.cc:729] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c5a2ce2b1148389b55f7570c243f01, State: Initialized, Role: FOLLOWER
I20260620 18:39:13.191210 13573 consensus_queue.cc:260] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.191344 13573 ts_tablet_manager.cc:1428] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:13.191543 13577 tablet_bootstrap.cc:492] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780: Bootstrap starting.
I20260620 18:39:13.191934 13574 tablet_bootstrap.cc:492] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072: No bootstrap required, opened a new log
I20260620 18:39:13.192039 13574 ts_tablet_manager.cc:1397] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:39:13.192050 13577 tablet_bootstrap.cc:654] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:13.192171 13574 raft_consensus.cc:348] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.192248 13574 raft_consensus.cc:374] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:13.192263 13574 raft_consensus.cc:729] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41c569c07254088a3c061f18a978072, State: Initialized, Role: FOLLOWER
I20260620 18:39:13.192322 13574 consensus_queue.cc:260] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.192477 13574 ts_tablet_manager.cc:1428] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:13.192706 13577 tablet_bootstrap.cc:492] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780: No bootstrap required, opened a new log
I20260620 18:39:13.192770 13577 ts_tablet_manager.cc:1397] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:13.192925 13577 raft_consensus.cc:348] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.192997 13577 raft_consensus.cc:374] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:13.193037 13577 raft_consensus.cc:729] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2794a8e11097434d99c6cf6fe061f780, State: Initialized, Role: FOLLOWER
I20260620 18:39:13.193089 13577 consensus_queue.cc:260] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.193251 13577 ts_tablet_manager.cc:1428] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:13.359787 13543 raft_consensus.cc:482] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:13.359910 13543 raft_consensus.cc:504] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.360119 13543 leader_election.cc:288] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2794a8e11097434d99c6cf6fe061f780 (127.10.250.66:39027), 44c5a2ce2b1148389b55f7570c243f01 (127.10.250.67:36527)
I20260620 18:39:13.360392 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a6b1fbb77c4c6e882e1b964173f235" candidate_uuid: "a41c569c07254088a3c061f18a978072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c5a2ce2b1148389b55f7570c243f01" is_pre_election: true
I20260620 18:39:13.360477 13410 raft_consensus.cc:2454] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41c569c07254088a3c061f18a978072 in term 0.
I20260620 18:39:13.360423 13269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a6b1fbb77c4c6e882e1b964173f235" candidate_uuid: "a41c569c07254088a3c061f18a978072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794a8e11097434d99c6cf6fe061f780" is_pre_election: true
I20260620 18:39:13.360564 13269 raft_consensus.cc:2454] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41c569c07254088a3c061f18a978072 in term 0.
I20260620 18:39:13.360702 13063 leader_election.cc:302] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c5a2ce2b1148389b55f7570c243f01, a41c569c07254088a3c061f18a978072; no voters:
I20260620 18:39:13.360823 13543 raft_consensus.cc:2781] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:13.360868 13543 raft_consensus.cc:482] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:13.360889 13543 raft_consensus.cc:3037] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:13.361608 13543 raft_consensus.cc:504] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.361759 13543 leader_election.cc:288] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [CANDIDATE]: Term 1 election: Requested vote from peers 2794a8e11097434d99c6cf6fe061f780 (127.10.250.66:39027), 44c5a2ce2b1148389b55f7570c243f01 (127.10.250.67:36527)
I20260620 18:39:13.361949 13269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a6b1fbb77c4c6e882e1b964173f235" candidate_uuid: "a41c569c07254088a3c061f18a978072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794a8e11097434d99c6cf6fe061f780"
I20260620 18:39:13.361970 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a6b1fbb77c4c6e882e1b964173f235" candidate_uuid: "a41c569c07254088a3c061f18a978072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c5a2ce2b1148389b55f7570c243f01"
I20260620 18:39:13.362040 13269 raft_consensus.cc:3037] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:13.362039 13410 raft_consensus.cc:3037] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:13.362905 13410 raft_consensus.cc:2454] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41c569c07254088a3c061f18a978072 in term 1.
I20260620 18:39:13.363140 13063 leader_election.cc:302] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c5a2ce2b1148389b55f7570c243f01, a41c569c07254088a3c061f18a978072; no voters:
I20260620 18:39:13.363281 13543 raft_consensus.cc:2781] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:13.363338 13543 raft_consensus.cc:686] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [term 1 LEADER]: Becoming Leader. State: Replica: a41c569c07254088a3c061f18a978072, State: Running, Role: LEADER
I20260620 18:39:13.363391 13269 raft_consensus.cc:2454] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41c569c07254088a3c061f18a978072 in term 1.
I20260620 18:39:13.363401 13543 consensus_queue.cc:237] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.364074 12972 catalog_manager.cc:5495] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 reported cstate change: term changed from 0 to 1, leader changed from <none> to a41c569c07254088a3c061f18a978072 (127.10.250.65). New cstate: current_term: 1 leader_uuid: "a41c569c07254088a3c061f18a978072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } health_report { overall_health: UNKNOWN } } }
I20260620 18:39:13.376792 13582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:13.374270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.10.250.65:42789)
0620 18:39:13.374450 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:13.374451 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:13.374458 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:39:13.374486 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:13.374487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:13.374497 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:13.374593 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:13.374778 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.375270 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.376035 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.376261 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.376338 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:13.376507 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:13.376548 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:13.376610 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:13.376671 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 18:39:13.376695 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":100,"threads_started":1}
I20260620 18:39:13.379024 13410 raft_consensus.cc:1261] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01 [term 1 FOLLOWER]: Refusing update from remote peer a41c569c07254088a3c061f18a978072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:13.379024 13269 raft_consensus.cc:1261] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Refusing update from remote peer a41c569c07254088a3c061f18a978072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:13.379359 13540 consensus_queue.cc:1035] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:13.379478 13548 consensus_queue.cc:1035] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:13.389331 12972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:13.389432 12972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:13.390002 12972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f: Sending DeleteTablet for 3 replicas of tablet f6a6b1fbb77c4c6e882e1b964173f235
I20260620 18:39:13.390308 13250 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a6b1fbb77c4c6e882e1b964173f235 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:13 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:44156
I20260620 18:39:13.390327 13390 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a6b1fbb77c4c6e882e1b964173f235 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:13 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:45928
I20260620 18:39:13.390435 13564 tablet_replica.cc:330] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780: stopping tablet replica
I20260620 18:39:13.390452 13109 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a6b1fbb77c4c6e882e1b964173f235 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:13 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:45586
I20260620 18:39:13.390568 13562 tablet_replica.cc:330] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072: stopping tablet replica
I20260620 18:39:13.390583 13563 tablet_replica.cc:330] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01: stopping tablet replica
I20260620 18:39:13.390640 13563 raft_consensus.cc:2229] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:13.390651 13562 raft_consensus.cc:2229] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:13.390692 13563 raft_consensus.cc:2258] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:13.390722 13562 raft_consensus.cc:2258] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:13.390766 13564 raft_consensus.cc:2229] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:13.390817 13564 raft_consensus.cc:2258] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:13.391053 13563 ts_tablet_manager.cc:1905] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:13.391059 13562 ts_tablet_manager.cc:1905] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:13.391032 12972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:39:13.391180 13564 ts_tablet_manager.cc:1905] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 18:39:13.391191 12972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:13.392391 13250 tablet_service.cc:1467] Processing CreateTablet for tablet 10b8a094f89b404eb16f436c89a3872e (DEFAULT_TABLE table=test-table [id=1ae4ee3162b74b1d83bff5a0906061b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:13.392491 13250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10b8a094f89b404eb16f436c89a3872e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:13.392762 13390 tablet_service.cc:1467] Processing CreateTablet for tablet 10b8a094f89b404eb16f436c89a3872e (DEFAULT_TABLE table=test-table [id=1ae4ee3162b74b1d83bff5a0906061b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:13.392843 13564 ts_tablet_manager.cc:1918] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:13.392891 13564 log.cc:1196] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/f6a6b1fbb77c4c6e882e1b964173f235
I20260620 18:39:13.392904 13563 ts_tablet_manager.cc:1918] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:13.392963 13563 log.cc:1196] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/f6a6b1fbb77c4c6e882e1b964173f235
I20260620 18:39:13.393069 13562 ts_tablet_manager.cc:1918] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:13.393112 13564 ts_tablet_manager.cc:1939] T f6a6b1fbb77c4c6e882e1b964173f235 P 2794a8e11097434d99c6cf6fe061f780: Deleting consensus metadata
I20260620 18:39:13.393121 13562 log.cc:1196] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-0/wal/wals/f6a6b1fbb77c4c6e882e1b964173f235
I20260620 18:39:13.393196 13563 ts_tablet_manager.cc:1939] T f6a6b1fbb77c4c6e882e1b964173f235 P 44c5a2ce2b1148389b55f7570c243f01: Deleting consensus metadata
I20260620 18:39:13.393325 13562 ts_tablet_manager.cc:1939] T f6a6b1fbb77c4c6e882e1b964173f235 P a41c569c07254088a3c061f18a978072: Deleting consensus metadata
I20260620 18:39:13.393366 13390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10b8a094f89b404eb16f436c89a3872e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:13.393721 13577 tablet_bootstrap.cc:492] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780: Bootstrap starting.
I20260620 18:39:13.393869 12959 catalog_manager.cc:4841] TS a41c569c07254088a3c061f18a978072 (127.10.250.65:42789): tablet f6a6b1fbb77c4c6e882e1b964173f235 (table test-table [id=6c405f1d9ae94d738d26032b63e59ac2]) successfully deleted
I20260620 18:39:13.394110 13109 tablet_service.cc:1467] Processing CreateTablet for tablet 10b8a094f89b404eb16f436c89a3872e (DEFAULT_TABLE table=test-table [id=1ae4ee3162b74b1d83bff5a0906061b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:13.394160 13577 tablet_bootstrap.cc:654] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:13.394198 13109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10b8a094f89b404eb16f436c89a3872e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:13.394644 13573 tablet_bootstrap.cc:492] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01: Bootstrap starting.
I20260620 18:39:13.394979 13573 tablet_bootstrap.cc:654] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:13.395345 13574 tablet_bootstrap.cc:492] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072: Bootstrap starting.
I20260620 18:39:13.395427 13577 tablet_bootstrap.cc:492] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780: No bootstrap required, opened a new log
I20260620 18:39:13.395467 13577 ts_tablet_manager.cc:1397] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:39:13.395591 12959 catalog_manager.cc:4841] TS 44c5a2ce2b1148389b55f7570c243f01 (127.10.250.67:36527): tablet f6a6b1fbb77c4c6e882e1b964173f235 (table test-table [id=6c405f1d9ae94d738d26032b63e59ac2]) successfully deleted
I20260620 18:39:13.395655 12959 catalog_manager.cc:4841] TS 2794a8e11097434d99c6cf6fe061f780 (127.10.250.66:39027): tablet f6a6b1fbb77c4c6e882e1b964173f235 (table test-table [id=6c405f1d9ae94d738d26032b63e59ac2]) successfully deleted
I20260620 18:39:13.395687 13574 tablet_bootstrap.cc:654] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:13.395656 13577 raft_consensus.cc:348] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.395740 13577 raft_consensus.cc:374] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:13.395761 13577 raft_consensus.cc:729] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2794a8e11097434d99c6cf6fe061f780, State: Initialized, Role: FOLLOWER
I20260620 18:39:13.395805 13577 consensus_queue.cc:260] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.395905 13577 ts_tablet_manager.cc:1428] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:13.396140 13573 tablet_bootstrap.cc:492] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01: No bootstrap required, opened a new log
I20260620 18:39:13.396195 13573 ts_tablet_manager.cc:1397] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:39:13.396363 13573 raft_consensus.cc:348] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.396440 13573 raft_consensus.cc:374] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:13.396464 13573 raft_consensus.cc:729] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c5a2ce2b1148389b55f7570c243f01, State: Initialized, Role: FOLLOWER
I20260620 18:39:13.396514 13573 consensus_queue.cc:260] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.396636 13573 ts_tablet_manager.cc:1428] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:13.397107 13574 tablet_bootstrap.cc:492] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072: No bootstrap required, opened a new log
I20260620 18:39:13.397204 13574 ts_tablet_manager.cc:1397] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:39:13.397317 13574 raft_consensus.cc:348] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.397395 13574 raft_consensus.cc:374] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:13.397420 13574 raft_consensus.cc:729] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41c569c07254088a3c061f18a978072, State: Initialized, Role: FOLLOWER
I20260620 18:39:13.397471 13574 consensus_queue.cc:260] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.397567 13574 ts_tablet_manager.cc:1428] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:13.711827 13557 raft_consensus.cc:482] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:13.711934 13557 raft_consensus.cc:504] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.712163 13557 leader_election.cc:288] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41c569c07254088a3c061f18a978072 (127.10.250.65:42789), 2794a8e11097434d99c6cf6fe061f780 (127.10.250.66:39027)
I20260620 18:39:13.712503 13130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b8a094f89b404eb16f436c89a3872e" candidate_uuid: "44c5a2ce2b1148389b55f7570c243f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41c569c07254088a3c061f18a978072" is_pre_election: true
I20260620 18:39:13.712531 13270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b8a094f89b404eb16f436c89a3872e" candidate_uuid: "44c5a2ce2b1148389b55f7570c243f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794a8e11097434d99c6cf6fe061f780" is_pre_election: true
I20260620 18:39:13.712621 13270 raft_consensus.cc:2454] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c5a2ce2b1148389b55f7570c243f01 in term 0.
I20260620 18:39:13.712627 13130 raft_consensus.cc:2454] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c5a2ce2b1148389b55f7570c243f01 in term 0.
I20260620 18:39:13.712819 13343 leader_election.cc:302] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c5a2ce2b1148389b55f7570c243f01, a41c569c07254088a3c061f18a978072; no voters:
I20260620 18:39:13.712921 13557 raft_consensus.cc:2781] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:13.712977 13557 raft_consensus.cc:482] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:13.713045 13557 raft_consensus.cc:3037] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:13.713831 13557 raft_consensus.cc:504] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.713979 13557 leader_election.cc:288] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [CANDIDATE]: Term 1 election: Requested vote from peers a41c569c07254088a3c061f18a978072 (127.10.250.65:42789), 2794a8e11097434d99c6cf6fe061f780 (127.10.250.66:39027)
I20260620 18:39:13.714283 13130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b8a094f89b404eb16f436c89a3872e" candidate_uuid: "44c5a2ce2b1148389b55f7570c243f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41c569c07254088a3c061f18a978072"
I20260620 18:39:13.714283 13270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b8a094f89b404eb16f436c89a3872e" candidate_uuid: "44c5a2ce2b1148389b55f7570c243f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794a8e11097434d99c6cf6fe061f780"
I20260620 18:39:13.714377 13270 raft_consensus.cc:3037] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:13.714377 13130 raft_consensus.cc:3037] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:13.715222 13270 raft_consensus.cc:2454] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c5a2ce2b1148389b55f7570c243f01 in term 1.
I20260620 18:39:13.715248 13130 raft_consensus.cc:2454] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c5a2ce2b1148389b55f7570c243f01 in term 1.
I20260620 18:39:13.715411 13343 leader_election.cc:302] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2794a8e11097434d99c6cf6fe061f780, 44c5a2ce2b1148389b55f7570c243f01; no voters:
I20260620 18:39:13.715529 13557 raft_consensus.cc:2781] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:13.715654 13557 raft_consensus.cc:686] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [term 1 LEADER]: Becoming Leader. State: Replica: 44c5a2ce2b1148389b55f7570c243f01, State: Running, Role: LEADER
I20260620 18:39:13.715751 13557 consensus_queue.cc:237] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } }
I20260620 18:39:13.716348 12973 catalog_manager.cc:5495] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44c5a2ce2b1148389b55f7570c243f01 (127.10.250.67). New cstate: current_term: 1 leader_uuid: "44c5a2ce2b1148389b55f7570c243f01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c5a2ce2b1148389b55f7570c243f01" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36527 } health_report { overall_health: HEALTHY } } }
I20260620 18:39:13.754619 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:13.752231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45946 (local address 127.10.250.67:36527)
0620 18:39:13.752371 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:13.752373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:13.752382 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:13.752402 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:13.752402 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:13.752413 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:13.752489 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:13.752610 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.753208 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.753960 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.754151 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.754215 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:13.754330 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:13.754358 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:13.754397 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:13.754444 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 18:39:13.754461 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260620 18:39:13.756930 13130 raft_consensus.cc:1261] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072 [term 1 FOLLOWER]: Refusing update from remote peer 44c5a2ce2b1148389b55f7570c243f01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:13.756929 13270 raft_consensus.cc:1261] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Refusing update from remote peer 44c5a2ce2b1148389b55f7570c243f01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:13.757179 13557 consensus_queue.cc:1035] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2794a8e11097434d99c6cf6fe061f780" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 39027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:13.757321 13589 consensus_queue.cc:1035] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41c569c07254088a3c061f18a978072" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:39:13.761906 13545 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 18:39:15.286442 12971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:15.286594 12971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:15.287468 12971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14fc285cb2da4c4ab0627bdfe9c21f1f: Sending DeleteTablet for 3 replicas of tablet 10b8a094f89b404eb16f436c89a3872e
I20260620 18:39:15.287729 13109 tablet_service.cc:1514] Processing DeleteTablet for tablet 10b8a094f89b404eb16f436c89a3872e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:15 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:45586
I20260620 18:39:15.287824 13250 tablet_service.cc:1514] Processing DeleteTablet for tablet 10b8a094f89b404eb16f436c89a3872e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:15 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:44156
I20260620 18:39:15.287935 13618 tablet_replica.cc:330] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072: stopping tablet replica
I20260620 18:39:15.287994 13618 raft_consensus.cc:2229] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:15.288023 13619 tablet_replica.cc:330] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780: stopping tablet replica
I20260620 18:39:15.288012 13390 tablet_service.cc:1514] Processing DeleteTablet for tablet 10b8a094f89b404eb16f436c89a3872e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:15 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:45928
I20260620 18:39:15.288067 13619 raft_consensus.cc:2229] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:15.288029 13618 raft_consensus.cc:2258] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:15.288093 13619 raft_consensus.cc:2258] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:15.288196 13620 tablet_replica.cc:330] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01: stopping tablet replica
I20260620 18:39:15.288338 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 13045
I20260620 18:39:15.288408 13618 ts_tablet_manager.cc:1905] T 10b8a094f89b404eb16f436c89a3872e P a41c569c07254088a3c061f18a978072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:15.289884 13619 ts_tablet_manager.cc:1905] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:15.291471 13620 raft_consensus.cc:2229] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:15.291586 13620 raft_consensus.cc:2258] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:15.291724 13619 ts_tablet_manager.cc:1918] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:39:15.291792 13619 log.cc:1196] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/10b8a094f89b404eb16f436c89a3872e
I20260620 18:39:15.292007 13619 ts_tablet_manager.cc:1939] T 10b8a094f89b404eb16f436c89a3872e P 2794a8e11097434d99c6cf6fe061f780: Deleting consensus metadata
I20260620 18:39:15.292464 12959 catalog_manager.cc:4841] TS 2794a8e11097434d99c6cf6fe061f780 (127.10.250.66:39027): tablet 10b8a094f89b404eb16f436c89a3872e (table test-table [id=1ae4ee3162b74b1d83bff5a0906061b3]) successfully deleted
I20260620 18:39:15.292495 13620 ts_tablet_manager.cc:1905] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:15.293781 13620 ts_tablet_manager.cc:1918] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:39:15.293828 13620 log.cc:1196] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/10b8a094f89b404eb16f436c89a3872e
I20260620 18:39:15.294045 13620 ts_tablet_manager.cc:1939] T 10b8a094f89b404eb16f436c89a3872e P 44c5a2ce2b1148389b55f7570c243f01: Deleting consensus metadata
I20260620 18:39:15.294406 12959 catalog_manager.cc:4841] TS 44c5a2ce2b1148389b55f7570c243f01 (127.10.250.67:36527): tablet 10b8a094f89b404eb16f436c89a3872e (table test-table [id=1ae4ee3162b74b1d83bff5a0906061b3]) successfully deleted
W20260620 18:39:15.295498 12959 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:42789 (error 108)
I20260620 18:39:15.295724 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 13185
I20260620 18:39:15.296413 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:15.295879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:42789 (local address 127.0.0.1:45630)
0620 18:39:15.296311 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:15.296364 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":56,"threads_started":1}
W20260620 18:39:15.296522 12959 catalog_manager.cc:4584] TS a41c569c07254088a3c061f18a978072 (127.10.250.65:42789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10b8a094f89b404eb16f436c89a3872e: Network error: Client connection negotiation failed: client connection to 127.10.250.65:42789: connect: Connection refused (error 111)
I20260620 18:39:15.301872 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 13325
I20260620 18:39:15.307544 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 12941
2026-06-20T18:39:15Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6195 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:15 dist-test-slave-wp6g krb5kdc[13625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:15 dist-test-slave-wp6g krb5kdc[13625](info): set up 2 sockets
Jun 20 18:39:15 dist-test-slave-wp6g krb5kdc[13625](info): commencing operation
krb5kdc: starting...
W20260620 18:39:16.358279 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:39:16 dist-test-slave-wp6g krb5kdc[13625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:39:16.478149 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:36571
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:36571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:16.558961 13638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:16.559170 13638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:16.559191 13638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:16.560470 13638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:16.560544 13638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:16.560570 13638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:16.560602 13638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:16.560631 13638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:16.562168 13638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:36571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:36571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:16.562464 13638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:16.562723 13638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:16.568368 13646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:16.568416 13644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:16.568493 13638 server_base.cc:1034] running on GCE node
W20260620 18:39:16.568452 13643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:16.568781 13638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:16.569099 13638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:16.570259 13638 hybrid_clock.cc:648] HybridClock initialized: now 1781980756570223 us; error 45 us; skew 500 ppm
Jun 20 18:39:16 dist-test-slave-wp6g krb5kdc[13625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980756, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:16.572980 13638 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:39:16.573477 13638 webserver.cc:460] Webserver started at http://127.10.250.126:44895/ using document root <none> and password file <none>
I20260620 18:39:16.573734 13638 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:16.573798 13638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:16.573930 13638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:16.575094 13638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "932b76ad9f5046649de31d7590903aeb"
format_stamp: "Formatted at 2026-06-20 18:39:16 on dist-test-slave-wp6g"
server_key: "12afcd6572a21c9536fc8c0460c43387"
server_key_iv: "3d5605bf36b31103083d83f431c5c5a9"
server_key_version: "encryptionkey@0"
I20260620 18:39:16.575471 13638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "932b76ad9f5046649de31d7590903aeb"
format_stamp: "Formatted at 2026-06-20 18:39:16 on dist-test-slave-wp6g"
server_key: "12afcd6572a21c9536fc8c0460c43387"
server_key_iv: "3d5605bf36b31103083d83f431c5c5a9"
server_key_version: "encryptionkey@0"
I20260620 18:39:16.576967 13638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:39:16.577770 13653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:16.577965 13638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:16.578071 13638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "932b76ad9f5046649de31d7590903aeb"
format_stamp: "Formatted at 2026-06-20 18:39:16 on dist-test-slave-wp6g"
server_key: "12afcd6572a21c9536fc8c0460c43387"
server_key_iv: "3d5605bf36b31103083d83f431c5c5a9"
server_key_version: "encryptionkey@0"
I20260620 18:39:16.578179 13638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:16.601807 13638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:16.604128 13638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:16.604305 13638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:16.608333 13638 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:36571
I20260620 18:39:16.608456 13705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:36571 every 8 connection(s)
I20260620 18:39:16.608781 13638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:16.609673 13706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:16.612073 13706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb: Bootstrap starting.
I20260620 18:39:16.612717 13706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:16.613037 13706 log.cc:824] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:16.613356 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 13638
I20260620 18:39:16.613476 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:16.613658 11241 external_mini_cluster.cc:1428] Setting key 3885e74f588836bf1cd6a62e4aee19ad
I20260620 18:39:16.613868 13706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb: No bootstrap required, opened a new log
I20260620 18:39:16.616175 13706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932b76ad9f5046649de31d7590903aeb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36571 } }
I20260620 18:39:16.616401 13706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:16.616451 13706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932b76ad9f5046649de31d7590903aeb, State: Initialized, Role: FOLLOWER
I20260620 18:39:16.616604 13706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932b76ad9f5046649de31d7590903aeb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36571 } }
I20260620 18:39:16.616704 13706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:16.616756 13706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:16.616806 13706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:16.617684 13706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932b76ad9f5046649de31d7590903aeb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36571 } }
I20260620 18:39:16.617825 13706 leader_election.cc:302] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 932b76ad9f5046649de31d7590903aeb; no voters:
I20260620 18:39:16.618002 13706 leader_election.cc:288] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:16.618076 13711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:16.618288 13711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [term 1 LEADER]: Becoming Leader. State: Replica: 932b76ad9f5046649de31d7590903aeb, State: Running, Role: LEADER
I20260620 18:39:16.618294 13706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:16.619124 13711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932b76ad9f5046649de31d7590903aeb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36571 } }
I20260620 18:39:16.619915 13713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 932b76ad9f5046649de31d7590903aeb. Latest consensus state: current_term: 1 leader_uuid: "932b76ad9f5046649de31d7590903aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932b76ad9f5046649de31d7590903aeb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36571 } } }
I20260620 18:39:16.619774 13712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "932b76ad9f5046649de31d7590903aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932b76ad9f5046649de31d7590903aeb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36571 } } }
I20260620 18:39:16.620095 13712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:16.620296 13713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb [sys.catalog]: This master's current role is: LEADER
W20260620 18:39:16.620350 13725 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:39:16.620422 13725 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:39:16.620703 13728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:16.620934 13728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 18:39:16 dist-test-slave-wp6g krb5kdc[13625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:16.622995 13728 catalog_manager.cc:1347] Generated new cluster ID: 3cf017c9cf0f4cb5b3b1bf4525b29de6
I20260620 18:39:16.623056 13728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:16.626684 13710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:16.614237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60314 (local address 127.10.250.126:36571)
0620 18:39:16.614460 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:16.614471 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:16.614510 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:39:16.615381 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:16.615392 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:16.615467 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:16.615639 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:16.616268 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.618059 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.619149 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.620106 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.620615 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:16.623334 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:16.623339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:16.623339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:16.623355 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:16.624984 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:16.625386 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:16.625390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:16.625391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:16.625491 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:16.625739 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:16.625740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:16.625740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:16.626002 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:16.626195 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:16.626265 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 18:39:16.626412 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":120,"threads_started":1}
I20260620 18:39:16.636554 13728 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:16.637290 13728 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:16.641865 13728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 932b76ad9f5046649de31d7590903aeb: Generated new TSK 0
I20260620 18:39:16.642069 13728 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:39:16.673837 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:36571
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:16.758955 13733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:16.759172 13733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:16.759210 13733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:16.760473 13733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:16.760520 13733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:16.760550 13733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:16.762144 13733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:36571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:16.762413 13733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:16.762708 13733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:16.768076 13738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:16.768076 13739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:16.768098 13741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:16.768529 13733 server_base.cc:1034] running on GCE node
I20260620 18:39:16.768718 13733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:16.768996 13733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:16.770197 13733 hybrid_clock.cc:648] HybridClock initialized: now 1781980756770151 us; error 32 us; skew 500 ppm
Jun 20 18:39:16 dist-test-slave-wp6g krb5kdc[13625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980756, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:16.772966 13733 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:39:16.773502 13733 webserver.cc:460] Webserver started at http://127.10.250.65:43071/ using document root <none> and password file <none>
I20260620 18:39:16.773813 13733 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:16.773892 13733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:16.774061 13733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:16.775279 13733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "034e5d6a91d34bb2a9efd9a3d6b29efe"
format_stamp: "Formatted at 2026-06-20 18:39:16 on dist-test-slave-wp6g"
server_key: "a1263eba01f4a18155895effd258be98"
server_key_iv: "e99287e406383fedde23fb08f96a41e5"
server_key_version: "encryptionkey@0"
I20260620 18:39:16.775660 13733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "034e5d6a91d34bb2a9efd9a3d6b29efe"
format_stamp: "Formatted at 2026-06-20 18:39:16 on dist-test-slave-wp6g"
server_key: "a1263eba01f4a18155895effd258be98"
server_key_iv: "e99287e406383fedde23fb08f96a41e5"
server_key_version: "encryptionkey@0"
I20260620 18:39:16.777108 13733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:39:16.778069 13748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:16.778319 13733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:16.778438 13733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "034e5d6a91d34bb2a9efd9a3d6b29efe"
format_stamp: "Formatted at 2026-06-20 18:39:16 on dist-test-slave-wp6g"
server_key: "a1263eba01f4a18155895effd258be98"
server_key_iv: "e99287e406383fedde23fb08f96a41e5"
server_key_version: "encryptionkey@0"
I20260620 18:39:16.778553 13733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:16.790184 13733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:16.792834 13733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:16.793081 13733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:16.793381 13733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:16.793774 13733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:16.793833 13733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:16.793875 13733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:16.793905 13733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:16.800892 13733 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:34041
I20260620 18:39:16.800930 13861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:34041 every 8 connection(s)
I20260620 18:39:16.801556 13733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:39:16 dist-test-slave-wp6g krb5kdc[13625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980756, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:16.809273 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 13733
I20260620 18:39:16.809376 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:16.809578 11241 external_mini_cluster.cc:1428] Setting key 8b0c14902bde8bab7fa374d5f87294b2
I20260620 18:39:16.814116 13710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:16.802457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:42479 (local address 127.10.250.126:36571)
0620 18:39:16.802583 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:16.802586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:16.802911 (+ 325us) server_negotiation.cc:407] Connection header received
0620 18:39:16.803379 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:16.803380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:16.803390 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:16.803453 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:16.804916 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.805440 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.806507 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.806771 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.806838 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:16.809396 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:16.809398 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:16.809398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:16.809409 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:16.812277 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:16.812809 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:16.812810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:16.812810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:16.812882 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:16.813228 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:16.813229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:16.813229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:16.813400 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:16.813475 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:16.813862 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 18:39:16.813973 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 18:39:16.814590 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:16.802471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:36571 (local address 127.10.250.65:42479)
0620 18:39:16.802720 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:16.802767 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:16.803268 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:16.803518 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:16.803521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:16.804360 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:16.804663 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:16.804672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.805551 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.805553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:16.806395 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:16.806398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.806884 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.806885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:16.807006 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:16.807637 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:16.807646 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:16.809204 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:16.812401 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:16.812414 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:16.812417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:16.812715 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:16.812985 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:16.812986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:16.812987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:16.813141 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:16.813496 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:16.813504 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:16.813692 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 18:39:16.813811 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 18:39:16.813970 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":115,"threads_started":1}
I20260620 18:39:16.815382 13862 heartbeater.cc:347] Connected to a master server at 127.10.250.126:36571
I20260620 18:39:16.815495 13862 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:16.815753 13862 heartbeater.cc:511] Master 127.10.250.126:36571 requested a full tablet report, sending...
I20260620 18:39:16.816668 13670 ts_manager.cc:194] Registered new tserver with Master: 034e5d6a91d34bb2a9efd9a3d6b29efe (127.10.250.65:34041)
I20260620 18:39:16.817369 13670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:42479
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:39:16.843691 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:36571
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:16.932741 13867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:16.932945 13867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:16.932971 13867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:16.934154 13867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:16.934195 13867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:16.934229 13867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:16.935741 13867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:36571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:16.935947 13867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:16.936131 13867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:16.941355 13873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:16.941344 13875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:16.941344 13872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:16.941457 13867 server_base.cc:1034] running on GCE node
I20260620 18:39:16.941783 13867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:16.942013 13867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:16.943176 13867 hybrid_clock.cc:648] HybridClock initialized: now 1781980756943159 us; error 26 us; skew 500 ppm
Jun 20 18:39:16 dist-test-slave-wp6g krb5kdc[13625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980756, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:16.945748 13867 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:39:16.946251 13867 webserver.cc:460] Webserver started at http://127.10.250.66:44441/ using document root <none> and password file <none>
I20260620 18:39:16.946550 13867 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:16.946626 13867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:16.946766 13867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:16.948000 13867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "2b90194b0de7444fbb5df8a3d9773340"
format_stamp: "Formatted at 2026-06-20 18:39:16 on dist-test-slave-wp6g"
server_key: "1377613d1bd0cf0e34b7cf8a294263ed"
server_key_iv: "933fa2f10b7c430f2861ffd8b8a73a3f"
server_key_version: "encryptionkey@0"
I20260620 18:39:16.948400 13867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "2b90194b0de7444fbb5df8a3d9773340"
format_stamp: "Formatted at 2026-06-20 18:39:16 on dist-test-slave-wp6g"
server_key: "1377613d1bd0cf0e34b7cf8a294263ed"
server_key_iv: "933fa2f10b7c430f2861ffd8b8a73a3f"
server_key_version: "encryptionkey@0"
I20260620 18:39:16.949893 13867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:16.950847 13882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:16.951161 13867 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:16.951294 13867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "2b90194b0de7444fbb5df8a3d9773340"
format_stamp: "Formatted at 2026-06-20 18:39:16 on dist-test-slave-wp6g"
server_key: "1377613d1bd0cf0e34b7cf8a294263ed"
server_key_iv: "933fa2f10b7c430f2861ffd8b8a73a3f"
server_key_version: "encryptionkey@0"
I20260620 18:39:16.951385 13867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:16.971055 13867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:16.973439 13867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:16.973640 13867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:16.973920 13867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:16.974267 13867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:16.974323 13867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:16.974366 13867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:16.974413 13867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:16.980321 13867 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:41259
I20260620 18:39:16.980383 13995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:41259 every 8 connection(s)
I20260620 18:39:16.980775 13867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:39:16 dist-test-slave-wp6g krb5kdc[13625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980756, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:16.990417 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 13867
I20260620 18:39:16.990568 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:16.990806 11241 external_mini_cluster.cc:1428] Setting key 395d4b1731fae5241e9de5a0036849c7
I20260620 18:39:16.990959 13710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:16.981588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:48311 (local address 127.10.250.126:36571)
0620 18:39:16.981651 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:16.981654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:16.981952 (+ 298us) server_negotiation.cc:407] Connection header received
0620 18:39:16.982471 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:16.982473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:16.982484 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:16.982609 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:16.983607 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.984114 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.985129 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.985339 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.985399 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:16.987934 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:16.987936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:16.987936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:16.987944 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:16.989425 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:16.989867 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:16.989868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:16.989869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:16.989925 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:16.990160 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:16.990160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:16.990161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:16.990292 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:16.990386 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:16.990767 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 18:39:16.990857 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 18:39:16.991191 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:16.981602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:36571 (local address 127.10.250.66:48311)
0620 18:39:16.981822 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:16.981861 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:16.982371 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:16.982684 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:16.982688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:16.983295 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:16.983496 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:16.983501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.984208 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.984209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:16.985047 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:16.985050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.985434 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.985435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:16.985551 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:16.986103 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:16.986113 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:16.987839 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:16.989524 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:16.989533 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:16.989535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:16.989791 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:16.989989 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:16.989990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:16.989990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:16.990101 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:16.990385 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:16.990393 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:16.990608 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 18:39:16.990745 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 18:39:16.990931 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":87,"threads_started":1}
I20260620 18:39:16.991727 13996 heartbeater.cc:347] Connected to a master server at 127.10.250.126:36571
I20260620 18:39:16.991848 13996 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:16.992048 13996 heartbeater.cc:511] Master 127.10.250.126:36571 requested a full tablet report, sending...
I20260620 18:39:16.992503 13670 ts_manager.cc:194] Registered new tserver with Master: 2b90194b0de7444fbb5df8a3d9773340 (127.10.250.66:41259)
I20260620 18:39:16.993041 13670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:48311
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:39:17.018388 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:36571
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:17.101358 14001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:17.101583 14001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:17.101608 14001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:17.102787 14001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:17.102824 14001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:17.102851 14001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:17.104328 14001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:36571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:17.104535 14001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:17.104714 14001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:17.109977 14009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:17.110073 14007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:17.110059 14006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:17.110320 14001 server_base.cc:1034] running on GCE node
I20260620 18:39:17.110582 14001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:17.110884 14001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:17.112036 14001 hybrid_clock.cc:648] HybridClock initialized: now 1781980757112019 us; error 43 us; skew 500 ppm
Jun 20 18:39:17 dist-test-slave-wp6g krb5kdc[13625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980757, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:17.114693 14001 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:39:17.115157 14001 webserver.cc:460] Webserver started at http://127.10.250.67:43485/ using document root <none> and password file <none>
I20260620 18:39:17.115403 14001 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:17.115473 14001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:17.115628 14001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:17.116704 14001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "baed480018ee4bc298e7c361da933d37"
format_stamp: "Formatted at 2026-06-20 18:39:17 on dist-test-slave-wp6g"
server_key: "5d2db6a3f282c6505dc641756bf64aa7"
server_key_iv: "2b7140007ffe7c70707e564987fa6657"
server_key_version: "encryptionkey@0"
I20260620 18:39:17.117084 14001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "baed480018ee4bc298e7c361da933d37"
format_stamp: "Formatted at 2026-06-20 18:39:17 on dist-test-slave-wp6g"
server_key: "5d2db6a3f282c6505dc641756bf64aa7"
server_key_iv: "2b7140007ffe7c70707e564987fa6657"
server_key_version: "encryptionkey@0"
I20260620 18:39:17.118446 14001 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:17.119295 14016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:17.119481 14001 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:39:17.119575 14001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "baed480018ee4bc298e7c361da933d37"
format_stamp: "Formatted at 2026-06-20 18:39:17 on dist-test-slave-wp6g"
server_key: "5d2db6a3f282c6505dc641756bf64aa7"
server_key_iv: "2b7140007ffe7c70707e564987fa6657"
server_key_version: "encryptionkey@0"
I20260620 18:39:17.119663 14001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:17.131285 14001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:17.133798 14001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:17.133977 14001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:17.134240 14001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:17.134651 14001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:17.134706 14001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:17.134755 14001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:17.134783 14001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:17.140815 14001 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:44355
I20260620 18:39:17.140893 14129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:44355 every 8 connection(s)
I20260620 18:39:17.141242 14001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:17.143584 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 14001
I20260620 18:39:17.143666 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:17.143859 11241 external_mini_cluster.cc:1428] Setting key 77079c89d8a8ec7a77ec6b5f41dc608d
Jun 20 18:39:17 dist-test-slave-wp6g krb5kdc[13625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980757, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:17.151734 13710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:17.142088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:36769 (local address 127.10.250.126:36571)
0620 18:39:17.142149 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:17.142151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:17.142359 (+ 208us) server_negotiation.cc:407] Connection header received
0620 18:39:17.143002 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:17.143003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:17.143016 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:17.143068 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:17.144194 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:17.144852 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:17.145879 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:17.146100 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:17.146189 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:17.148581 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:17.148583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:17.148583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:17.148592 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:17.150090 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:17.150633 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:17.150634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:17.150635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:17.150701 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:17.151001 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:17.151001 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:17.151002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:17.151132 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:17.151212 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:17.151529 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 18:39:17.151632 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 18:39:17.151950 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:17.142077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:36571 (local address 127.10.250.67:36769)
0620 18:39:17.142235 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:17.142274 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:17.142881 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:17.143145 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:17.143149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:17.143805 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:17.144096 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:17.144103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:17.144931 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:17.144932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:17.145779 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:17.145781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:17.146203 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:17.146204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:17.146351 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:17.146947 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:17.146956 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:17.148462 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:17.150198 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:17.150209 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:17.150238 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:17.150507 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:17.150780 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:17.150781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:17.150781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:17.150923 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:17.151219 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:17.151227 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:17.151385 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 18:39:17.151492 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 18:39:17.151677 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260620 18:39:17.152527 14130 heartbeater.cc:347] Connected to a master server at 127.10.250.126:36571
I20260620 18:39:17.152640 14130 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:17.152837 14130 heartbeater.cc:511] Master 127.10.250.126:36571 requested a full tablet report, sending...
I20260620 18:39:17.153254 13670 ts_manager.cc:194] Registered new tserver with Master: baed480018ee4bc298e7c361da933d37 (127.10.250.67:44355)
I20260620 18:39:17.153650 13670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:36769
I20260620 18:39:17.156049 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:39:17.167352 13710 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:17.166067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.10.250.126:36571)
0620 18:39:17.166133 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:17.166135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:17.166320 (+ 185us) server_negotiation.cc:407] Connection header received
0620 18:39:17.166353 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:17.166355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:17.166365 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:17.166414 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:17.167253 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60318: BlockingRecv error: recv got EOF from 127.0.0.1:60318 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:39:17.168509 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 13733
I20260620 18:39:17.173275 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 13867
I20260620 18:39:17.178164 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 14001
I20260620 18:39:17.183594 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 13638
2026-06-20T18:39:17Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1876 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:17 dist-test-slave-wp6g krb5kdc[14142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:17 dist-test-slave-wp6g krb5kdc[14142](info): set up 2 sockets
Jun 20 18:39:17 dist-test-slave-wp6g krb5kdc[14142](info): commencing operation
krb5kdc: starting...
W20260620 18:39:18.242394 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:39:18 dist-test-slave-wp6g krb5kdc[14142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:39:18.361650 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:39729
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:39729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:18.443789 14155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:18.444011 14155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:18.444036 14155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:18.445189 14155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:18.445225 14155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:18.445240 14155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:18.445253 14155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:18.445266 14155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:18.446758 14155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:39729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:39729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:18.446954 14155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:18.447132 14155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:18.451948 14163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:18.451961 14161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:18.451962 14160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:18.452037 14155 server_base.cc:1034] running on GCE node
I20260620 18:39:18.452438 14155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:18.452735 14155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:18.453925 14155 hybrid_clock.cc:648] HybridClock initialized: now 1781980758453908 us; error 26 us; skew 500 ppm
Jun 20 18:39:18 dist-test-slave-wp6g krb5kdc[14142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980758, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:18.459915 14155 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:39:18.460426 14155 webserver.cc:460] Webserver started at http://127.10.250.126:35073/ using document root <none> and password file <none>
I20260620 18:39:18.460676 14155 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:18.460753 14155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:18.460901 14155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:18.462010 14155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "26d2363f00b546ebb1a191ff0096128a"
format_stamp: "Formatted at 2026-06-20 18:39:18 on dist-test-slave-wp6g"
server_key: "142404f5cdd8c1fc35986e9ab969a120"
server_key_iv: "4bbd50175a210654d68a2c8ba3a3b722"
server_key_version: "encryptionkey@0"
I20260620 18:39:18.462368 14155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "26d2363f00b546ebb1a191ff0096128a"
format_stamp: "Formatted at 2026-06-20 18:39:18 on dist-test-slave-wp6g"
server_key: "142404f5cdd8c1fc35986e9ab969a120"
server_key_iv: "4bbd50175a210654d68a2c8ba3a3b722"
server_key_version: "encryptionkey@0"
I20260620 18:39:18.463774 14155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:18.464545 14170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:18.464715 14155 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:39:18.464823 14155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "26d2363f00b546ebb1a191ff0096128a"
format_stamp: "Formatted at 2026-06-20 18:39:18 on dist-test-slave-wp6g"
server_key: "142404f5cdd8c1fc35986e9ab969a120"
server_key_iv: "4bbd50175a210654d68a2c8ba3a3b722"
server_key_version: "encryptionkey@0"
I20260620 18:39:18.464905 14155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:18.475083 14155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:18.477382 14155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:18.477550 14155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:18.481832 14155 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:39729
I20260620 18:39:18.481868 14222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:39729 every 8 connection(s)
I20260620 18:39:18.482365 14155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:18.483121 14223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:18.485345 14223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a: Bootstrap starting.
I20260620 18:39:18.485893 14223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:18.486133 14223 log.cc:824] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:18.486588 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 14155
I20260620 18:39:18.486714 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:18.486845 14223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a: No bootstrap required, opened a new log
I20260620 18:39:18.486876 11241 external_mini_cluster.cc:1428] Setting key 3e0e2edfe7f2ebd61fb244b093438b0a
I20260620 18:39:18.489161 14223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d2363f00b546ebb1a191ff0096128a" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39729 } }
I20260620 18:39:18.489322 14223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:18.489367 14223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d2363f00b546ebb1a191ff0096128a, State: Initialized, Role: FOLLOWER
I20260620 18:39:18.489507 14223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d2363f00b546ebb1a191ff0096128a" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39729 } }
I20260620 18:39:18.489606 14223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:18.489651 14223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:18.489699 14223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:18.490229 14223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d2363f00b546ebb1a191ff0096128a" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39729 } }
I20260620 18:39:18.490350 14223 leader_election.cc:302] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26d2363f00b546ebb1a191ff0096128a; no voters:
I20260620 18:39:18.490571 14223 leader_election.cc:288] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:18.490588 14228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:18.490722 14228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [term 1 LEADER]: Becoming Leader. State: Replica: 26d2363f00b546ebb1a191ff0096128a, State: Running, Role: LEADER
I20260620 18:39:18.490844 14223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:18.490832 14228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d2363f00b546ebb1a191ff0096128a" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39729 } }
I20260620 18:39:18.491156 14228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26d2363f00b546ebb1a191ff0096128a. Latest consensus state: current_term: 1 leader_uuid: "26d2363f00b546ebb1a191ff0096128a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d2363f00b546ebb1a191ff0096128a" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39729 } } }
I20260620 18:39:18.491165 14229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26d2363f00b546ebb1a191ff0096128a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d2363f00b546ebb1a191ff0096128a" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39729 } } }
I20260620 18:39:18.491245 14228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:18.491321 14229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:18.491636 14233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:18.492182 14233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:18.493993 14233 catalog_manager.cc:1347] Generated new cluster ID: 00a242fd1af448618f6eb0772ff9f2fa
I20260620 18:39:18.494065 14233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:39:18 dist-test-slave-wp6g krb5kdc[14142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:18.500124 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:18.487368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39794 (local address 127.10.250.126:39729)
0620 18:39:18.487560 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:18.487574 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:18.487601 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:39:18.489153 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:18.489161 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:18.489217 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:18.489405 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:18.490143 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.491663 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.492805 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.493413 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.493523 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:18.496411 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:18.496415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:18.496415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:18.496427 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:18.498368 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:18.498880 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:18.498883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:18.498883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:18.499005 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:18.499222 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:18.499223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:18.499224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:18.499514 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:18.499674 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:18.499778 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 18:39:18.499908 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260620 18:39:18.501300 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:39:18.514379 14233 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:18.515736 14233 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:18.522624 14233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26d2363f00b546ebb1a191ff0096128a: Generated new TSK 0
I20260620 18:39:18.522838 14233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:39:18.523366 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:18.518842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39800 (local address 127.10.250.126:39729)
0620 18:39:18.518907 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:18.518909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:18.519135 (+ 226us) server_negotiation.cc:407] Connection header received
0620 18:39:18.519204 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:18.519205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:18.519217 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:18.519275 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:18.519725 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.520465 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.521546 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.521857 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.521924 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:18.522836 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:18.522837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:18.522837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:18.523089 (+ 252us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:18.523107 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:18.523114 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:18.523170 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:18.523264 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 18:39:18.523289 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:39:18.525624 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 14155
2026-06-20T18:39:18Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1338 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:18 dist-test-slave-wp6g krb5kdc[14256](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:18 dist-test-slave-wp6g krb5kdc[14256](info): set up 2 sockets
Jun 20 18:39:18 dist-test-slave-wp6g krb5kdc[14256](info): commencing operation
krb5kdc: starting...
W20260620 18:39:19.562600 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:39:19 dist-test-slave-wp6g krb5kdc[14256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:39:19.685209 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:39287
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:39287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:19.767330 14269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:19.767589 14269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:19.767632 14269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:19.768942 14269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:19.769009 14269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:19.769033 14269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:19.769075 14269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:19.769098 14269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:19.770743 14269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:39287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:39287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:19.771070 14269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:19.771355 14269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:19.777735 14277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:19.777731 14275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:19.777733 14274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:19.778316 14269 server_base.cc:1034] running on GCE node
I20260620 18:39:19.778623 14269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:19.778939 14269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:19.780143 14269 hybrid_clock.cc:648] HybridClock initialized: now 1781980759780128 us; error 28 us; skew 500 ppm
Jun 20 18:39:19 dist-test-slave-wp6g krb5kdc[14256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980759, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:19.782797 14269 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:39:19.783269 14269 webserver.cc:460] Webserver started at http://127.10.250.126:36353/ using document root <none> and password file <none>
I20260620 18:39:19.783506 14269 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:19.783576 14269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:19.783730 14269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:19.784857 14269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "fa6640f191354e9a9095f0f363f55948"
format_stamp: "Formatted at 2026-06-20 18:39:19 on dist-test-slave-wp6g"
server_key: "26d00bef5a328147a7523559475d9cad"
server_key_iv: "9a104d34268307c72a92a81d54ef1ec6"
server_key_version: "encryptionkey@0"
I20260620 18:39:19.785236 14269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "fa6640f191354e9a9095f0f363f55948"
format_stamp: "Formatted at 2026-06-20 18:39:19 on dist-test-slave-wp6g"
server_key: "26d00bef5a328147a7523559475d9cad"
server_key_iv: "9a104d34268307c72a92a81d54ef1ec6"
server_key_version: "encryptionkey@0"
I20260620 18:39:19.786792 14269 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:39:19.787645 14284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:19.787865 14269 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:19.787984 14269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "fa6640f191354e9a9095f0f363f55948"
format_stamp: "Formatted at 2026-06-20 18:39:19 on dist-test-slave-wp6g"
server_key: "26d00bef5a328147a7523559475d9cad"
server_key_iv: "9a104d34268307c72a92a81d54ef1ec6"
server_key_version: "encryptionkey@0"
I20260620 18:39:19.788066 14269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:19.795279 14269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:19.797219 14269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:19.797386 14269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:19.801605 14269 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:39287
I20260620 18:39:19.801637 14336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:39287 every 8 connection(s)
I20260620 18:39:19.801998 14269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:19.803090 14337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:19.805352 14337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948: Bootstrap starting.
I20260620 18:39:19.805994 14337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:19.806291 14337 log.cc:824] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:19.807180 14337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948: No bootstrap required, opened a new log
I20260620 18:39:19.808759 14337 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6640f191354e9a9095f0f363f55948" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39287 } }
I20260620 18:39:19.808925 14337 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:19.808949 14337 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6640f191354e9a9095f0f363f55948, State: Initialized, Role: FOLLOWER
I20260620 18:39:19.809108 14337 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6640f191354e9a9095f0f363f55948" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39287 } }
I20260620 18:39:19.809173 14337 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:19.809194 14337 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:19.809222 14337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:19.810056 14337 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6640f191354e9a9095f0f363f55948" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39287 } }
I20260620 18:39:19.810145 14337 leader_election.cc:302] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa6640f191354e9a9095f0f363f55948; no voters:
I20260620 18:39:19.810310 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 14269
I20260620 18:39:19.810376 14337 leader_election.cc:288] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:19.810398 14340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:19.810426 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:19.810539 14340 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [term 1 LEADER]: Becoming Leader. State: Replica: fa6640f191354e9a9095f0f363f55948, State: Running, Role: LEADER
I20260620 18:39:19.810642 11241 external_mini_cluster.cc:1428] Setting key 0cfa21c57018ab6d8d781f736d77b687
I20260620 18:39:19.810634 14340 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6640f191354e9a9095f0f363f55948" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39287 } }
I20260620 18:39:19.810705 14337 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:19.810933 14340 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa6640f191354e9a9095f0f363f55948. Latest consensus state: current_term: 1 leader_uuid: "fa6640f191354e9a9095f0f363f55948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6640f191354e9a9095f0f363f55948" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39287 } } }
I20260620 18:39:19.811015 14340 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:19.810940 14341 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa6640f191354e9a9095f0f363f55948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6640f191354e9a9095f0f363f55948" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39287 } } }
I20260620 18:39:19.811062 14341 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:19.811574 14344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:19.812125 14344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:19.813746 14344 catalog_manager.cc:1347] Generated new cluster ID: fa612df6697e4753892158053e931a08
I20260620 18:39:19.813814 14344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:39:19 dist-test-slave-wp6g krb5kdc[14256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:19.821264 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:19.811293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.10.250.126:39287)
0620 18:39:19.811491 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:19.811500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:19.811539 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:39:19.811725 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:19.811733 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:19.811800 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:19.811956 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:19.812781 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.814053 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.814965 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.815372 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.815489 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.817820 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:19.817825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:19.817825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:19.817845 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:19.819607 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:19.819999 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:19.820002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:19.820002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:19.820118 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:19.820337 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:19.820337 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:19.820338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:19.820604 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:19.820787 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:19.820886 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 18:39:19.821035 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
I20260620 18:39:19.830420 14344 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:19.831040 14344 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:19.839856 14344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa6640f191354e9a9095f0f363f55948: Generated new TSK 0
I20260620 18:39:19.840090 14344 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:39:19.869903 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:39287
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:19.952960 14363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:19.953179 14363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:19.953205 14363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:19.954499 14363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:19.954784 14363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:19.954908 14363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:19.956682 14363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:39287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:19.956974 14363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:19.957481 14363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:19.963128 14369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:19.963135 14368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:19.963138 14371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:19.963373 14363 server_base.cc:1034] running on GCE node
I20260620 18:39:19.963657 14363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:19.963922 14363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:19.965094 14363 hybrid_clock.cc:648] HybridClock initialized: now 1781980759965066 us; error 46 us; skew 500 ppm
Jun 20 18:39:19 dist-test-slave-wp6g krb5kdc[14256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980759, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:19.967810 14363 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:39:19.968288 14363 webserver.cc:460] Webserver started at http://127.10.250.65:45531/ using document root <none> and password file <none>
I20260620 18:39:19.968534 14363 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:19.968602 14363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:19.968750 14363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:19.969894 14363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "2924ce05913a47f0b15e9f64dbf8f0fb"
format_stamp: "Formatted at 2026-06-20 18:39:19 on dist-test-slave-wp6g"
server_key: "9010532b62e0bf3f4351edcdd17cfac0"
server_key_iv: "16e3a28060953ff2c871588a06e6e501"
server_key_version: "encryptionkey@0"
I20260620 18:39:19.970259 14363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "2924ce05913a47f0b15e9f64dbf8f0fb"
format_stamp: "Formatted at 2026-06-20 18:39:19 on dist-test-slave-wp6g"
server_key: "9010532b62e0bf3f4351edcdd17cfac0"
server_key_iv: "16e3a28060953ff2c871588a06e6e501"
server_key_version: "encryptionkey@0"
I20260620 18:39:19.971659 14363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:19.972419 14378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:19.972595 14363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:19.972685 14363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "2924ce05913a47f0b15e9f64dbf8f0fb"
format_stamp: "Formatted at 2026-06-20 18:39:19 on dist-test-slave-wp6g"
server_key: "9010532b62e0bf3f4351edcdd17cfac0"
server_key_iv: "16e3a28060953ff2c871588a06e6e501"
server_key_version: "encryptionkey@0"
I20260620 18:39:19.972786 14363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:19.999709 14363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:20.002228 14363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:20.002408 14363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:20.002703 14363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:20.003038 14363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:20.003094 14363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:20.003140 14363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:20.003170 14363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:20.009413 14363 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:40387
I20260620 18:39:20.009461 14491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:40387 every 8 connection(s)
I20260620 18:39:20.009804 14363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:20.015285 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 14363
I20260620 18:39:20.015417 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:20.015614 11241 external_mini_cluster.cc:1428] Setting key ba3a790148ca9515697bc7e7fb56d0ea
Jun 20 18:39:20 dist-test-slave-wp6g krb5kdc[14256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980759, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:20.020141 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.010625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:53295 (local address 127.10.250.126:39287)
0620 18:39:20.010797 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:20.010799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:20.010950 (+ 151us) server_negotiation.cc:407] Connection header received
0620 18:39:20.011456 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:20.011457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:20.011470 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:20.011524 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:20.012722 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.013249 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.014272 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.014464 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.014569 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.017078 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:20.017080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:20.017080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:20.017089 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:20.018672 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:20.019117 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:20.019118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:20.019118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:20.019179 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:20.019459 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:20.019460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:20.019460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:20.019578 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:20.019654 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:20.019941 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 18:39:20.020044 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 18:39:20.020351 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.010561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:39287 (local address 127.10.250.65:53295)
0620 18:39:20.010812 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:20.010857 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:20.011353 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:20.011610 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:20.011613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:20.012425 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:20.012631 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:20.012635 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.013351 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.013352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:20.014183 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:20.014185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.014578 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.014580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:20.014704 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.015255 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:20.015267 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:20.016967 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:20.018774 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:20.018784 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:20.018787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:20.019029 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:20.019247 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:20.019248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:20.019249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:20.019380 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:20.019644 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:20.019651 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:20.019805 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 18:39:20.019919 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 18:39:20.020089 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
I20260620 18:39:20.020836 14492 heartbeater.cc:347] Connected to a master server at 127.10.250.126:39287
I20260620 18:39:20.020961 14492 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:20.021203 14492 heartbeater.cc:511] Master 127.10.250.126:39287 requested a full tablet report, sending...
I20260620 18:39:20.021767 14301 ts_manager.cc:194] Registered new tserver with Master: 2924ce05913a47f0b15e9f64dbf8f0fb (127.10.250.65:40387)
I20260620 18:39:20.022430 14301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:53295
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:39:20.041420 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:39287
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:20.124590 14497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:20.124828 14497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:20.124856 14497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:20.126137 14497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:20.126183 14497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:20.126214 14497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:20.127777 14497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:39287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:20.127996 14497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:20.128189 14497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:20.133401 14505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:20.133401 14503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:20.133376 14502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:20.133826 14497 server_base.cc:1034] running on GCE node
I20260620 18:39:20.134035 14497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:20.134310 14497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:20.135454 14497 hybrid_clock.cc:648] HybridClock initialized: now 1781980760135439 us; error 41 us; skew 500 ppm
Jun 20 18:39:20 dist-test-slave-wp6g krb5kdc[14256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980760, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:20.138121 14497 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:39:20.138720 14497 webserver.cc:460] Webserver started at http://127.10.250.66:37031/ using document root <none> and password file <none>
I20260620 18:39:20.138973 14497 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:20.139042 14497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:20.139189 14497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:20.140287 14497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "417b80740ffd45cbbc2056fdbe92bcd0"
format_stamp: "Formatted at 2026-06-20 18:39:20 on dist-test-slave-wp6g"
server_key: "2d059dd67f26a1fcf46faf98ed812938"
server_key_iv: "d8f7f58d8d8dc86f9893a45ef1b6ae09"
server_key_version: "encryptionkey@0"
I20260620 18:39:20.140681 14497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "417b80740ffd45cbbc2056fdbe92bcd0"
format_stamp: "Formatted at 2026-06-20 18:39:20 on dist-test-slave-wp6g"
server_key: "2d059dd67f26a1fcf46faf98ed812938"
server_key_iv: "d8f7f58d8d8dc86f9893a45ef1b6ae09"
server_key_version: "encryptionkey@0"
I20260620 18:39:20.142166 14497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:20.143069 14512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:20.143244 14497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:39:20.143363 14497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "417b80740ffd45cbbc2056fdbe92bcd0"
format_stamp: "Formatted at 2026-06-20 18:39:20 on dist-test-slave-wp6g"
server_key: "2d059dd67f26a1fcf46faf98ed812938"
server_key_iv: "d8f7f58d8d8dc86f9893a45ef1b6ae09"
server_key_version: "encryptionkey@0"
I20260620 18:39:20.143443 14497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:20.151747 14497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:20.153787 14497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:20.153976 14497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:20.154212 14497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:20.154592 14497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:20.154651 14497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:20.154695 14497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:20.154721 14497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:20.160831 14497 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:40043
I20260620 18:39:20.160868 14625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:40043 every 8 connection(s)
I20260620 18:39:20.161211 14497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:20.166469 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 14497
I20260620 18:39:20.166623 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:20.166833 11241 external_mini_cluster.cc:1428] Setting key 072fb7fc550c8bd6de4585b2c7ab0312
Jun 20 18:39:20 dist-test-slave-wp6g krb5kdc[14256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980760, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:20.171802 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.161984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:38371 (local address 127.10.250.126:39287)
0620 18:39:20.162029 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:20.162031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:20.162233 (+ 202us) server_negotiation.cc:407] Connection header received
0620 18:39:20.162869 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:20.162871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:20.162884 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:20.162931 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:20.163954 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.164473 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.165498 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.165705 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.165784 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.168291 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:20.168292 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:20.168293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:20.168301 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:20.170141 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:20.170671 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:20.170673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:20.170673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:20.170742 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:20.170998 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:20.170999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:20.171000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:20.171161 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:20.171231 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:20.171577 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 18:39:20.171686 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 18:39:20.171840 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.161947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:39287 (local address 127.10.250.66:38371)
0620 18:39:20.162089 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:20.162144 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:20.162732 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:20.163000 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:20.163005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:20.163632 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:20.163844 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:20.163849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.164575 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.164576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:20.165403 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:20.165404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.165793 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.165794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:20.165918 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.166480 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:20.166489 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:20.168187 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:20.170244 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:20.170252 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:20.170255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:20.170598 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:20.170815 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:20.170816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:20.170816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:20.170926 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:20.171226 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:20.171230 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:20.171373 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 18:39:20.171516 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 18:39:20.171657 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260620 18:39:20.172721 14626 heartbeater.cc:347] Connected to a master server at 127.10.250.126:39287
I20260620 18:39:20.172842 14626 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:20.173069 14626 heartbeater.cc:511] Master 127.10.250.126:39287 requested a full tablet report, sending...
I20260620 18:39:20.173547 14301 ts_manager.cc:194] Registered new tserver with Master: 417b80740ffd45cbbc2056fdbe92bcd0 (127.10.250.66:40043)
I20260620 18:39:20.173949 14301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:38371
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:39:20.193986 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:39287
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:20.277324 14631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:20.277545 14631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:20.277583 14631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:20.278731 14631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:20.278798 14631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:20.278846 14631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:20.280292 14631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:39287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:20.280558 14631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:20.280797 14631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:20.285740 14637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:20.285770 14636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:20.285740 14639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:20.286068 14631 server_base.cc:1034] running on GCE node
I20260620 18:39:20.286295 14631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:20.286629 14631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:20.287791 14631 hybrid_clock.cc:648] HybridClock initialized: now 1781980760287775 us; error 45 us; skew 500 ppm
Jun 20 18:39:20 dist-test-slave-wp6g krb5kdc[14256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980760, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:20.290582 14631 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:39:20.291074 14631 webserver.cc:460] Webserver started at http://127.10.250.67:33417/ using document root <none> and password file <none>
I20260620 18:39:20.291352 14631 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:20.291424 14631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:20.291589 14631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:20.292737 14631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "4416ff8c772d4f1f88d045799f4c08a7"
format_stamp: "Formatted at 2026-06-20 18:39:20 on dist-test-slave-wp6g"
server_key: "9c0cf502d004083fba0d5a3ec1d29bc6"
server_key_iv: "4c5cb937c3816b82a414d002652044f3"
server_key_version: "encryptionkey@0"
I20260620 18:39:20.293113 14631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "4416ff8c772d4f1f88d045799f4c08a7"
format_stamp: "Formatted at 2026-06-20 18:39:20 on dist-test-slave-wp6g"
server_key: "9c0cf502d004083fba0d5a3ec1d29bc6"
server_key_iv: "4c5cb937c3816b82a414d002652044f3"
server_key_version: "encryptionkey@0"
I20260620 18:39:20.294579 14631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 18:39:20.295400 14646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:20.295593 14631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:20.295696 14631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "4416ff8c772d4f1f88d045799f4c08a7"
format_stamp: "Formatted at 2026-06-20 18:39:20 on dist-test-slave-wp6g"
server_key: "9c0cf502d004083fba0d5a3ec1d29bc6"
server_key_iv: "4c5cb937c3816b82a414d002652044f3"
server_key_version: "encryptionkey@0"
I20260620 18:39:20.295787 14631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:20.323563 14631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:20.326006 14631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:20.326229 14631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:20.326553 14631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:20.326913 14631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:20.326968 14631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:20.327013 14631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:20.327039 14631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:20.334005 14631 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:33845
I20260620 18:39:20.334048 14759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:33845 every 8 connection(s)
I20260620 18:39:20.334470 14631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:20.340374 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 14631
I20260620 18:39:20.340468 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:20.340658 11241 external_mini_cluster.cc:1428] Setting key b626df28fa2e221590277014ebf8b1ec
Jun 20 18:39:20 dist-test-slave-wp6g krb5kdc[14256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980760, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:20.344908 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.335372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:52147 (local address 127.10.250.126:39287)
0620 18:39:20.335436 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:20.335438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:20.335663 (+ 225us) server_negotiation.cc:407] Connection header received
0620 18:39:20.336355 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:20.336356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:20.336368 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:20.336436 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:20.337558 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.338071 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.339221 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.339441 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.339507 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.342060 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:20.342061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:20.342061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:20.342069 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:20.343555 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:20.343971 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:20.343972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:20.343972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:20.344029 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:20.344250 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:20.344250 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:20.344250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:20.344363 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:20.344416 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:20.344767 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 18:39:20.344838 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 18:39:20.345268 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.335352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:39287 (local address 127.10.250.67:52147)
0620 18:39:20.335510 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:20.335556 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:20.336059 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:20.336525 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:20.336528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:20.337221 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:20.337439 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:20.337444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.338197 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.338197 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:20.339136 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:20.339138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.339552 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.339553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:20.339675 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.340281 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:20.340290 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:20.341972 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:20.343635 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:20.343642 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:20.343644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:20.343904 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:20.344091 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:20.344091 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:20.344092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:20.344195 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:20.344449 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:20.344456 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:20.344621 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 18:39:20.344731 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 18:39:20.344887 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260620 18:39:20.345819 14760 heartbeater.cc:347] Connected to a master server at 127.10.250.126:39287
I20260620 18:39:20.345945 14760 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:20.346166 14760 heartbeater.cc:511] Master 127.10.250.126:39287 requested a full tablet report, sending...
I20260620 18:39:20.346655 14301 ts_manager.cc:194] Registered new tserver with Master: 4416ff8c772d4f1f88d045799f4c08a7 (127.10.250.67:33845)
I20260620 18:39:20.347045 14301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:52147
I20260620 18:39:20.352821 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:39:20 dist-test-slave-wp6g krb5kdc[14256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980760, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 18:39:20 dist-test-slave-wp6g krb5kdc[14256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980760, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:20.377146 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.369393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50276 (local address 127.10.250.126:39287)
0620 18:39:20.369465 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:20.369468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:20.369628 (+ 160us) server_negotiation.cc:407] Connection header received
0620 18:39:20.369674 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:20.369675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:20.369685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:20.369736 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:20.370377 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.370938 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.371674 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.371858 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.371907 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.374265 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:20.374285 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:20.374286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:20.374293 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:20.375873 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:20.376224 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:20.376225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:20.376225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:20.376284 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:20.376565 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:20.376565 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:20.376566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:20.376725 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:20.376819 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:20.376947 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 18:39:20.377017 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 18:39:20.377432 14301 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50276
I20260620 18:39:20.377919 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 14363
I20260620 18:39:20.383172 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 14497
I20260620 18:39:20.387660 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 14631
I20260620 18:39:20.392179 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 14269
2026-06-20T18:39:20Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1868 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:20 dist-test-slave-wp6g krb5kdc[14772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:20 dist-test-slave-wp6g krb5kdc[14772](info): set up 2 sockets
Jun 20 18:39:20 dist-test-slave-wp6g krb5kdc[14772](info): commencing operation
krb5kdc: starting...
W20260620 18:39:21.430603 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:39:21 dist-test-slave-wp6g krb5kdc[14772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:39:21.557914 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:40175
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:40175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:21.641029 14785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:21.641291 14785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:21.641331 14785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:21.642493 14785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:21.642619 14785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:21.642645 14785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:21.642666 14785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:21.642694 14785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:21.644301 14785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:40175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:40175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:21.644595 14785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:21.644831 14785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:21.650063 14791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:21.650058 14793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:21.650099 14785 server_base.cc:1034] running on GCE node
W20260620 18:39:21.650074 14790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:21.650710 14785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:21.650987 14785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:21.652169 14785 hybrid_clock.cc:648] HybridClock initialized: now 1781980761652139 us; error 39 us; skew 500 ppm
Jun 20 18:39:21 dist-test-slave-wp6g krb5kdc[14772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980761, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:21.654857 14785 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:39:21.655341 14785 webserver.cc:460] Webserver started at http://127.10.250.126:36457/ using document root <none> and password file <none>
I20260620 18:39:21.655603 14785 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:21.655673 14785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:21.655815 14785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:21.656896 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "3e6ed750e2e94e21816e11d8ff9239f8"
format_stamp: "Formatted at 2026-06-20 18:39:21 on dist-test-slave-wp6g"
server_key: "89e17a07248434f0474e3fa6a944f78f"
server_key_iv: "2a2927b529183b53d477d6199998f72d"
server_key_version: "encryptionkey@0"
I20260620 18:39:21.657259 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "3e6ed750e2e94e21816e11d8ff9239f8"
format_stamp: "Formatted at 2026-06-20 18:39:21 on dist-test-slave-wp6g"
server_key: "89e17a07248434f0474e3fa6a944f78f"
server_key_iv: "2a2927b529183b53d477d6199998f72d"
server_key_version: "encryptionkey@0"
I20260620 18:39:21.658697 14785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:39:21.659487 14800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:21.659669 14785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:21.659771 14785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "3e6ed750e2e94e21816e11d8ff9239f8"
format_stamp: "Formatted at 2026-06-20 18:39:21 on dist-test-slave-wp6g"
server_key: "89e17a07248434f0474e3fa6a944f78f"
server_key_iv: "2a2927b529183b53d477d6199998f72d"
server_key_version: "encryptionkey@0"
I20260620 18:39:21.659852 14785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:21.667657 14785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:21.670042 14785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:21.670218 14785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:21.674774 14785 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:40175
I20260620 18:39:21.674813 14852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:40175 every 8 connection(s)
I20260620 18:39:21.675366 14785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:21.676155 14853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:21.678822 14853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8: Bootstrap starting.
I20260620 18:39:21.679476 14853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:21.679816 14853 log.cc:824] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:21.680711 14853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8: No bootstrap required, opened a new log
I20260620 18:39:21.682411 14853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ed750e2e94e21816e11d8ff9239f8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 40175 } }
I20260620 18:39:21.682647 14853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:21.682677 14853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6ed750e2e94e21816e11d8ff9239f8, State: Initialized, Role: FOLLOWER
I20260620 18:39:21.682842 14853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ed750e2e94e21816e11d8ff9239f8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 40175 } }
I20260620 18:39:21.682945 14853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:21.682994 14853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:21.683049 14853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:21.683063 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 14785
I20260620 18:39:21.683167 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:21.683352 11241 external_mini_cluster.cc:1428] Setting key a3cb502d0eae1eda6d64158c836edda5
I20260620 18:39:21.684429 14853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ed750e2e94e21816e11d8ff9239f8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 40175 } }
I20260620 18:39:21.684553 14853 leader_election.cc:302] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e6ed750e2e94e21816e11d8ff9239f8; no voters:
I20260620 18:39:21.684731 14853 leader_election.cc:288] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:21.684796 14858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:21.684975 14858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [term 1 LEADER]: Becoming Leader. State: Replica: 3e6ed750e2e94e21816e11d8ff9239f8, State: Running, Role: LEADER
I20260620 18:39:21.685021 14853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:21.685068 14858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ed750e2e94e21816e11d8ff9239f8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 40175 } }
I20260620 18:39:21.685412 14859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e6ed750e2e94e21816e11d8ff9239f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ed750e2e94e21816e11d8ff9239f8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 40175 } } }
I20260620 18:39:21.685511 14859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:21.685663 14860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e6ed750e2e94e21816e11d8ff9239f8. Latest consensus state: current_term: 1 leader_uuid: "3e6ed750e2e94e21816e11d8ff9239f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ed750e2e94e21816e11d8ff9239f8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 40175 } } }
I20260620 18:39:21.685720 14860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:21.685840 14864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:21.686376 14864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:21.688369 14864 catalog_manager.cc:1347] Generated new cluster ID: 15143d7e7aec44eb8bb27e333c7afd40
I20260620 18:39:21.688426 14864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:39:21 dist-test-slave-wp6g krb5kdc[14772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:21.695518 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:21.683982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.10.250.126:40175)
0620 18:39:21.684209 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:21.684220 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:21.684265 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:39:21.684393 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:21.684401 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:21.684463 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:21.684618 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:21.685610 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.687298 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.688325 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.688858 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.688926 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:21.691491 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:21.691496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:21.691496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:21.691509 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:21.693795 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:21.694208 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:21.694212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:21.694212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:21.694315 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:21.694555 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:21.694556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:21.694556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:21.694838 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:21.695018 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:21.695116 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 18:39:21.695264 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":110,"threads_started":1}
I20260620 18:39:21.702211 14864 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:21.702903 14864 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:21.707969 14864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e6ed750e2e94e21816e11d8ff9239f8: Generated new TSK 0
I20260620 18:39:21.708194 14864 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:39:21.746330 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:40175
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:21.854287 14879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:21.854589 14879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:21.854647 14879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:21.855863 14879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:21.855935 14879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:21.855994 14879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:21.857602 14879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:40175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:21.857909 14879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:21.858269 14879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:21.864327 14884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:21.864416 14887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:21.864526 14879 server_base.cc:1034] running on GCE node
W20260620 18:39:21.864416 14885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:21.864892 14879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:21.865159 14879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:21.866312 14879 hybrid_clock.cc:648] HybridClock initialized: now 1781980761866298 us; error 30 us; skew 500 ppm
Jun 20 18:39:21 dist-test-slave-wp6g krb5kdc[14772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980761, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:21.869078 14879 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:39:21.869583 14879 webserver.cc:460] Webserver started at http://127.10.250.65:33353/ using document root <none> and password file <none>
I20260620 18:39:21.869843 14879 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:21.869916 14879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:21.870069 14879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:21.871330 14879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "d81d04dfff774f9391c77676a3387757"
format_stamp: "Formatted at 2026-06-20 18:39:21 on dist-test-slave-wp6g"
server_key: "1a73f8c0408f761edd32dcb60e4bd720"
server_key_iv: "e411da8a8170cfe798d05eefbe4af8fe"
server_key_version: "encryptionkey@0"
I20260620 18:39:21.871737 14879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "d81d04dfff774f9391c77676a3387757"
format_stamp: "Formatted at 2026-06-20 18:39:21 on dist-test-slave-wp6g"
server_key: "1a73f8c0408f761edd32dcb60e4bd720"
server_key_iv: "e411da8a8170cfe798d05eefbe4af8fe"
server_key_version: "encryptionkey@0"
I20260620 18:39:21.873328 14879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:39:21.874333 14894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:21.874550 14879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:21.874655 14879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "d81d04dfff774f9391c77676a3387757"
format_stamp: "Formatted at 2026-06-20 18:39:21 on dist-test-slave-wp6g"
server_key: "1a73f8c0408f761edd32dcb60e4bd720"
server_key_iv: "e411da8a8170cfe798d05eefbe4af8fe"
server_key_version: "encryptionkey@0"
I20260620 18:39:21.874720 14879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:21.895293 14879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:21.897457 14879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:21.897576 14879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:21.897765 14879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:21.898066 14879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:21.898098 14879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:21.898123 14879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:21.898134 14879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:21.904057 14879 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:45427
I20260620 18:39:21.904120 15007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:45427 every 8 connection(s)
I20260620 18:39:21.904448 14879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:39:21 dist-test-slave-wp6g krb5kdc[14772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980761, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:21.911841 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 14879
I20260620 18:39:21.911953 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:21.912142 11241 external_mini_cluster.cc:1428] Setting key 3059d2ea6aa55c34f718f69c2461fd0a
I20260620 18:39:21.915006 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:21.905270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:38065 (local address 127.10.250.126:40175)
0620 18:39:21.905349 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:21.905352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:21.905565 (+ 213us) server_negotiation.cc:407] Connection header received
0620 18:39:21.906562 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:21.906563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:21.906579 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:21.906640 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:21.907268 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.907767 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.908897 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.909103 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.909162 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:21.911705 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:21.911707 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:21.911707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:21.911720 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:21.913425 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:21.913886 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:21.913887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:21.913887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:21.913958 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:21.914222 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:21.914223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:21.914224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:21.914384 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:21.914455 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:21.914809 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 18:39:21.914910 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:39:21.915236 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:21.905254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:40175 (local address 127.10.250.65:38065)
0620 18:39:21.905435 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:21.905474 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:21.906418 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:21.906710 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:21.906713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:21.906989 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:21.907182 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:21.907187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.907882 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.907883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:21.908796 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:21.908800 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.909248 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.909249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:21.909367 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:21.909915 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:21.909924 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:21.911592 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:21.913513 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:21.913522 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:21.913525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:21.913793 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:21.914027 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:21.914028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:21.914028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:21.914149 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:21.914450 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:21.914455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:21.914649 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 18:39:21.914768 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 18:39:21.914969 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
I20260620 18:39:21.915840 15008 heartbeater.cc:347] Connected to a master server at 127.10.250.126:40175
I20260620 18:39:21.915957 15008 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:21.916198 15008 heartbeater.cc:511] Master 127.10.250.126:40175 requested a full tablet report, sending...
I20260620 18:39:21.916781 14817 ts_manager.cc:194] Registered new tserver with Master: d81d04dfff774f9391c77676a3387757 (127.10.250.65:45427)
I20260620 18:39:21.917447 14817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:38065
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:39:21.941694 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:40175
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:22.024814 15013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:22.025048 15013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:22.025074 15013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:22.026325 15013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:22.026365 15013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:22.026392 15013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:22.027977 15013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:40175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:22.028182 15013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:22.028362 15013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:22.033540 15018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:22.033540 15021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:22.033619 15019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:22.033568 15013 server_base.cc:1034] running on GCE node
I20260620 18:39:22.033998 15013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:22.034281 15013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:22.035465 15013 hybrid_clock.cc:648] HybridClock initialized: now 1781980762035433 us; error 52 us; skew 500 ppm
Jun 20 18:39:22 dist-test-slave-wp6g krb5kdc[14772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980762, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:22.038251 15013 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:39:22.038745 15013 webserver.cc:460] Webserver started at http://127.10.250.66:38903/ using document root <none> and password file <none>
I20260620 18:39:22.038955 15013 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:22.038996 15013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:22.039090 15013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:22.040110 15013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "300e85ad8dab48848ed8ddd4ad8da321"
format_stamp: "Formatted at 2026-06-20 18:39:22 on dist-test-slave-wp6g"
server_key: "49f82e18c61b8fa79d3a3ceaa54c8894"
server_key_iv: "0cba9922022d5cc740f1965f6158591f"
server_key_version: "encryptionkey@0"
I20260620 18:39:22.040422 15013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "300e85ad8dab48848ed8ddd4ad8da321"
format_stamp: "Formatted at 2026-06-20 18:39:22 on dist-test-slave-wp6g"
server_key: "49f82e18c61b8fa79d3a3ceaa54c8894"
server_key_iv: "0cba9922022d5cc740f1965f6158591f"
server_key_version: "encryptionkey@0"
I20260620 18:39:22.041719 15013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:22.042433 15028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:22.042614 15013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:22.042718 15013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "300e85ad8dab48848ed8ddd4ad8da321"
format_stamp: "Formatted at 2026-06-20 18:39:22 on dist-test-slave-wp6g"
server_key: "49f82e18c61b8fa79d3a3ceaa54c8894"
server_key_iv: "0cba9922022d5cc740f1965f6158591f"
server_key_version: "encryptionkey@0"
I20260620 18:39:22.042801 15013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:22.064404 15013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:22.066371 15013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:22.066592 15013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:22.066862 15013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:22.067180 15013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:22.067234 15013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:22.067279 15013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:22.067313 15013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:22.073076 15013 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:46285
I20260620 18:39:22.073118 15141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:46285 every 8 connection(s)
I20260620 18:39:22.073472 15013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:22.077134 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 15013
I20260620 18:39:22.077224 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:22.077433 11241 external_mini_cluster.cc:1428] Setting key 63d20432ec31a58db71016c08f66a2be
Jun 20 18:39:22 dist-test-slave-wp6g krb5kdc[14772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980762, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:22.084414 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:22.074167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:35087 (local address 127.10.250.126:40175)
0620 18:39:22.074240 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:22.074242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:22.074488 (+ 246us) server_negotiation.cc:407] Connection header received
0620 18:39:22.075546 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:22.075547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:22.075561 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:22.075613 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:22.076267 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.076775 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.078121 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.078463 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.078561 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:22.080810 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:22.080812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:22.080812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:22.080820 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:22.082797 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:22.083289 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:22.083290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:22.083291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:22.083362 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:22.083663 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:22.083665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:22.083665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:22.083827 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:22.083908 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:22.084192 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 18:39:22.084303 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:39:22.084507 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:22.074170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:40175 (local address 127.10.250.66:35087)
0620 18:39:22.074340 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:22.074377 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:22.075412 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:22.075687 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:22.075692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:22.075961 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:22.076159 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:22.076163 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.076872 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.076873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:22.078036 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:22.078038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.078573 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.078574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:22.078741 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:22.079241 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:22.079253 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:22.080722 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:22.082891 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:22.082898 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:22.082900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:22.083202 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:22.083460 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:22.083462 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:22.083462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:22.083580 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:22.083909 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:22.083915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:22.084042 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 18:39:22.084141 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:39:22.084298 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260620 18:39:22.085119 15142 heartbeater.cc:347] Connected to a master server at 127.10.250.126:40175
I20260620 18:39:22.085233 15142 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:22.085470 15142 heartbeater.cc:511] Master 127.10.250.126:40175 requested a full tablet report, sending...
I20260620 18:39:22.085942 14817 ts_manager.cc:194] Registered new tserver with Master: 300e85ad8dab48848ed8ddd4ad8da321 (127.10.250.66:46285)
I20260620 18:39:22.086496 14817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:35087
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:39:22.104846 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:40175
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:22.189039 15147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:22.189257 15147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:22.189282 15147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:22.190466 15147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:22.190562 15147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:22.190629 15147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:22.192212 15147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:40175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:22.192466 15147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:22.192709 15147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:22.197923 15153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:22.197965 15155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:22.197933 15152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:22.198300 15147 server_base.cc:1034] running on GCE node
I20260620 18:39:22.198589 15147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:22.198853 15147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:22.200026 15147 hybrid_clock.cc:648] HybridClock initialized: now 1781980762200013 us; error 32 us; skew 500 ppm
Jun 20 18:39:22 dist-test-slave-wp6g krb5kdc[14772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980762, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:22.203207 15147 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:39:22.203709 15147 webserver.cc:460] Webserver started at http://127.10.250.67:35215/ using document root <none> and password file <none>
I20260620 18:39:22.203967 15147 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:22.204039 15147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:22.204186 15147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:22.205272 15147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "b0406e368cda4922a57eed3098f4b105"
format_stamp: "Formatted at 2026-06-20 18:39:22 on dist-test-slave-wp6g"
server_key: "cfc1d6cf1d7222d430f1f1b54c1d6bbc"
server_key_iv: "b719b31d7336af6e6e3125f1847c4407"
server_key_version: "encryptionkey@0"
I20260620 18:39:22.205674 15147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "b0406e368cda4922a57eed3098f4b105"
format_stamp: "Formatted at 2026-06-20 18:39:22 on dist-test-slave-wp6g"
server_key: "cfc1d6cf1d7222d430f1f1b54c1d6bbc"
server_key_iv: "b719b31d7336af6e6e3125f1847c4407"
server_key_version: "encryptionkey@0"
I20260620 18:39:22.207232 15147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:22.208297 15162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:22.208624 15147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:22.208719 15147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "b0406e368cda4922a57eed3098f4b105"
format_stamp: "Formatted at 2026-06-20 18:39:22 on dist-test-slave-wp6g"
server_key: "cfc1d6cf1d7222d430f1f1b54c1d6bbc"
server_key_iv: "b719b31d7336af6e6e3125f1847c4407"
server_key_version: "encryptionkey@0"
I20260620 18:39:22.208818 15147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:22.220381 15147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:22.222252 15147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:22.222416 15147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:22.222720 15147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:22.223057 15147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:22.223112 15147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:22.223155 15147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:22.223181 15147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:22.229290 15147 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:41605
I20260620 18:39:22.229341 15275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:41605 every 8 connection(s)
I20260620 18:39:22.229724 15147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
Jun 20 18:39:22 dist-test-slave-wp6g krb5kdc[14772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980762, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:22.239308 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:22.230498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:59589 (local address 127.10.250.126:40175)
0620 18:39:22.230597 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:22.230599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:22.230825 (+ 226us) server_negotiation.cc:407] Connection header received
0620 18:39:22.231809 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:22.231811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:22.231824 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:22.231875 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:22.232588 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.233103 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.234145 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.234357 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.234443 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:22.236604 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:22.236606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:22.236606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:22.236613 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:22.237856 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:22.238339 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:22.238340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:22.238340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:22.238396 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:22.238714 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:22.238715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:22.238715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:22.238843 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:22.238896 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:22.239172 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 18:39:22.239238 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:39:22.239506 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:22.230477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:40175 (local address 127.10.250.67:59589)
0620 18:39:22.230671 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:22.230723 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:22.231677 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:22.231949 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:22.231954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:22.232229 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:22.232478 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:22.232483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.233203 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.233204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:22.234038 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:22.234040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.234452 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.234453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:22.234621 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:22.235087 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:22.235095 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:22.236518 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:22.237930 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:22.237935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:22.237937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:22.238251 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:22.238460 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:22.238461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:22.238461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:22.238651 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:22.238913 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:22.238917 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:22.239041 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 18:39:22.239126 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 18:39:22.239279 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":121,"threads_started":1}
I20260620 18:39:22.239920 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 15147
I20260620 18:39:22.240020 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:22.240002 15276 heartbeater.cc:347] Connected to a master server at 127.10.250.126:40175
I20260620 18:39:22.240096 15276 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:22.240195 11241 external_mini_cluster.cc:1428] Setting key e5ebfce5375808fe1adbdb9f66374196
I20260620 18:39:22.240314 15276 heartbeater.cc:511] Master 127.10.250.126:40175 requested a full tablet report, sending...
I20260620 18:39:22.240677 14817 ts_manager.cc:194] Registered new tserver with Master: b0406e368cda4922a57eed3098f4b105 (127.10.250.67:41605)
I20260620 18:39:22.241093 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:39:22.241103 14817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:59589
Jun 20 18:39:22 dist-test-slave-wp6g krb5kdc[14772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:39:22 dist-test-slave-wp6g krb5kdc[14772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.65@KRBTEST.COM
I20260620 18:39:22.264380 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:22.256082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48024 (local address 127.10.250.65:45427)
0620 18:39:22.256283 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:22.256289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:22.256300 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:39:22.256335 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:22.256337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:22.256351 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:22.256435 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:22.256992 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.257728 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.258480 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.258852 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.258944 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:22.261210 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:22.261219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:22.261220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:22.261230 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:22.262970 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:22.263500 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:22.263503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:22.263503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:22.263582 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:22.263759 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:22.263760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:22.263760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:22.263912 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:22.264071 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:22.264173 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 18:39:22.264261 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
Jun 20 18:39:22 dist-test-slave-wp6g krb5kdc[14772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:22.272492 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:22.265708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55630 (local address 127.10.250.126:40175)
0620 18:39:22.265763 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:22.265765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:22.265775 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:39:22.265841 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:22.265842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:22.265853 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:22.265903 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:22.266380 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.266933 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.267635 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.267845 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.267929 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:22.270049 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:22.270050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:22.270050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:22.270058 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:22.271440 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:22.271791 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:22.271792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:22.271793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:22.271855 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:22.272025 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:22.272026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:22.272026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:22.272172 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:22.272247 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:22.272360 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 18:39:22.272425 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 18:39:22.272833 14817 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55630
I20260620 18:39:22.273243 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 14879
I20260620 18:39:22.278375 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 15013
I20260620 18:39:22.282470 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 15147
I20260620 18:39:22.286832 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 14785
2026-06-20T18:39:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1894 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:22 dist-test-slave-wp6g krb5kdc[15288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:22 dist-test-slave-wp6g krb5kdc[15288](info): set up 2 sockets
Jun 20 18:39:22 dist-test-slave-wp6g krb5kdc[15288](info): commencing operation
krb5kdc: starting...
W20260620 18:39:23.326676 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:39:23 dist-test-slave-wp6g krb5kdc[15288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:39:23.449432 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:41313
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:35471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:41313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:23.530170 15301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:23.530436 15301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:23.530480 15301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:23.531673 15301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:23.531742 15301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:23.531766 15301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:23.531788 15301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:23.531826 15301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:23.533350 15301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:35471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:41313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:41313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:23.533639 15301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:23.533885 15301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:23.538666 15309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:23.538736 15307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:23.538743 15301 server_base.cc:1034] running on GCE node
W20260620 18:39:23.538671 15306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:23.539143 15301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:23.539443 15301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:23.540606 15301 hybrid_clock.cc:648] HybridClock initialized: now 1781980763540599 us; error 34 us; skew 500 ppm
Jun 20 18:39:23 dist-test-slave-wp6g krb5kdc[15288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980763, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:23.546336 15301 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:39:23.546782 15301 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:23.546852 15301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:23.547016 15301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:23.548115 15301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "52cd0660ee0e412b8dbdbd3b21e2902c"
format_stamp: "Formatted at 2026-06-20 18:39:23 on dist-test-slave-wp6g"
server_key: "a32b1f9b46e488aec9ae7b4bc226ab34"
server_key_iv: "80d87446f39ab121ba94293e786537ec"
server_key_version: "encryptionkey@0"
I20260620 18:39:23.548485 15301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "52cd0660ee0e412b8dbdbd3b21e2902c"
format_stamp: "Formatted at 2026-06-20 18:39:23 on dist-test-slave-wp6g"
server_key: "a32b1f9b46e488aec9ae7b4bc226ab34"
server_key_iv: "80d87446f39ab121ba94293e786537ec"
server_key_version: "encryptionkey@0"
I20260620 18:39:23.549805 15301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:39:23.550629 15315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:23.550974 15301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:23.551106 15301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "52cd0660ee0e412b8dbdbd3b21e2902c"
format_stamp: "Formatted at 2026-06-20 18:39:23 on dist-test-slave-wp6g"
server_key: "a32b1f9b46e488aec9ae7b4bc226ab34"
server_key_iv: "80d87446f39ab121ba94293e786537ec"
server_key_version: "encryptionkey@0"
I20260620 18:39:23.551203 15301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:23.559569 15301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:23.561430 15301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:23.561596 15301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:23.565758 15301 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:41313
I20260620 18:39:23.565786 15367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:41313 every 8 connection(s)
I20260620 18:39:23.566077 15301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:23.567107 15368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:23.569355 15368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c: Bootstrap starting.
I20260620 18:39:23.569895 15368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:23.570138 15368 log.cc:824] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:23.570817 15368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c: No bootstrap required, opened a new log
I20260620 18:39:23.572422 15368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cd0660ee0e412b8dbdbd3b21e2902c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41313 } }
I20260620 18:39:23.572589 15368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:23.572634 15368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52cd0660ee0e412b8dbdbd3b21e2902c, State: Initialized, Role: FOLLOWER
I20260620 18:39:23.572830 15368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cd0660ee0e412b8dbdbd3b21e2902c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41313 } }
I20260620 18:39:23.572930 15368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:23.572974 15368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:23.573024 15368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:23.573899 15368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cd0660ee0e412b8dbdbd3b21e2902c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41313 } }
I20260620 18:39:23.574018 15368 leader_election.cc:302] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52cd0660ee0e412b8dbdbd3b21e2902c; no voters:
I20260620 18:39:23.574213 15368 leader_election.cc:288] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:23.574291 15371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:23.574422 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 15301
I20260620 18:39:23.574506 15371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [term 1 LEADER]: Becoming Leader. State: Replica: 52cd0660ee0e412b8dbdbd3b21e2902c, State: Running, Role: LEADER
I20260620 18:39:23.574563 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:23.574563 15368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:23.574628 15371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cd0660ee0e412b8dbdbd3b21e2902c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41313 } }
I20260620 18:39:23.574755 11241 external_mini_cluster.cc:1428] Setting key 890135b16ccea284e3845161e80c811e
I20260620 18:39:23.575009 15373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52cd0660ee0e412b8dbdbd3b21e2902c. Latest consensus state: current_term: 1 leader_uuid: "52cd0660ee0e412b8dbdbd3b21e2902c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cd0660ee0e412b8dbdbd3b21e2902c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41313 } } }
I20260620 18:39:23.575083 15373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:23.575496 15378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:23.575686 15372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52cd0660ee0e412b8dbdbd3b21e2902c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cd0660ee0e412b8dbdbd3b21e2902c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41313 } } }
I20260620 18:39:23.575841 15372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:23.576095 15378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:23.577816 15378 catalog_manager.cc:1347] Generated new cluster ID: f1eef2b7fc94488faefd06248ae42dd3
I20260620 18:39:23.577868 15378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:39:23 dist-test-slave-wp6g krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:23.587584 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:23.575369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.10.250.126:41313)
0620 18:39:23.576192 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:23.576202 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:23.576236 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:39:23.576379 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:23.576387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:23.576460 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:23.576604 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:23.577349 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.578752 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.579859 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.580490 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.580593 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:23.583547 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:23.583554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:23.583554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:23.583573 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:23.585686 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:23.586136 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:23.586141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:23.586141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:23.586283 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:23.586502 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:23.586503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:23.586503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:23.586924 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:23.587112 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:23.587189 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 18:39:23.587345 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":75,"threads_started":1}
I20260620 18:39:23.590593 15378 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:23.591365 15378 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:23.601606 15378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c: Generated new TSK 0
I20260620 18:39:23.601845 15378 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:39:23.634299 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:41313
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:35471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:23.717825 15394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:23.718086 15394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:23.718127 15394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:23.719378 15394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:23.719450 15394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:23.719498 15394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:23.721087 15394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:35471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:41313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:23.721380 15394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:23.721621 15394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:23.726656 15400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:23.726668 15402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:23.726706 15399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:23.726912 15394 server_base.cc:1034] running on GCE node
I20260620 18:39:23.727113 15394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:23.727346 15394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:23.728495 15394 hybrid_clock.cc:648] HybridClock initialized: now 1781980763728483 us; error 29 us; skew 500 ppm
Jun 20 18:39:23 dist-test-slave-wp6g krb5kdc[15288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980763, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:23.731055 15394 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:39:23.731408 15394 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:23.731482 15394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:23.731635 15394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:23.732720 15394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "15e851f018b044dbba7bfd70878f8f96"
format_stamp: "Formatted at 2026-06-20 18:39:23 on dist-test-slave-wp6g"
server_key: "bb7dbe31a22726b37019514c91771154"
server_key_iv: "19a95c5d68fb29b250266b0cb30aaa5e"
server_key_version: "encryptionkey@0"
I20260620 18:39:23.733100 15394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "15e851f018b044dbba7bfd70878f8f96"
format_stamp: "Formatted at 2026-06-20 18:39:23 on dist-test-slave-wp6g"
server_key: "bb7dbe31a22726b37019514c91771154"
server_key_iv: "19a95c5d68fb29b250266b0cb30aaa5e"
server_key_version: "encryptionkey@0"
I20260620 18:39:23.734416 15394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:39:23.735219 15408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:23.735422 15394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:23.735528 15394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "15e851f018b044dbba7bfd70878f8f96"
format_stamp: "Formatted at 2026-06-20 18:39:23 on dist-test-slave-wp6g"
server_key: "bb7dbe31a22726b37019514c91771154"
server_key_iv: "19a95c5d68fb29b250266b0cb30aaa5e"
server_key_version: "encryptionkey@0"
I20260620 18:39:23.735605 15394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:23.751541 15394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:23.753553 15394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:23.753724 15394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:23.753993 15394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:23.754428 15394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:23.754485 15394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:23.754522 15394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:23.754637 15394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:23.760476 15394 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:39275
I20260620 18:39:23.760515 15521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:39275 every 8 connection(s)
I20260620 18:39:23.760809 15394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:39:23 dist-test-slave-wp6g krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980763, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:23.769790 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 15394
I20260620 18:39:23.769913 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:23.770103 11241 external_mini_cluster.cc:1428] Setting key 9157941b880d0c995a337b66bb5d3b7e
I20260620 18:39:23.771667 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:23.761588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:47497 (local address 127.10.250.126:41313)
0620 18:39:23.761675 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:23.761677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:23.761824 (+ 147us) server_negotiation.cc:407] Connection header received
0620 18:39:23.762276 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:23.762277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:23.762291 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:23.762341 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:23.763294 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.763787 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.764815 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.765028 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.765094 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:23.767802 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:23.767804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:23.767804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:23.767815 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:23.769721 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:23.770392 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:23.770393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:23.770393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:23.770454 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:23.770793 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:23.770794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:23.770794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:23.770928 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:23.770976 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:23.771426 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 18:39:23.771526 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:39:23.771785 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:23.761541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41313 (local address 127.10.250.65:47497)
0620 18:39:23.761698 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:23.761735 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:23.762195 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:23.762412 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:23.762415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:23.762703 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:23.763202 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:23.763206 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.763887 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.763888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:23.764706 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:23.764708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.765110 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.765111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:23.765209 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:23.765789 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:23.765797 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:23.767696 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:23.769820 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:23.769827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:23.769829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:23.770291 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:23.770568 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:23.770569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:23.770569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:23.770708 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:23.771025 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:23.771031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:23.771207 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 18:39:23.771346 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 18:39:23.771524 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260620 18:39:23.772776 15522 heartbeater.cc:347] Connected to a master server at 127.10.250.126:41313
I20260620 18:39:23.772907 15522 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:23.773072 15522 heartbeater.cc:511] Master 127.10.250.126:41313 requested a full tablet report, sending...
I20260620 18:39:23.773684 15332 ts_manager.cc:194] Registered new tserver with Master: 15e851f018b044dbba7bfd70878f8f96 (127.10.250.65:39275)
I20260620 18:39:23.774729 15332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:47497
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:39:23.798171 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:41313
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:35471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:23.879791 15527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:23.880044 15527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:23.880086 15527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:23.881282 15527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:23.881363 15527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:23.881413 15527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:23.882974 15527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:35471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:41313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:23.883244 15527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:23.883499 15527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:23.888717 15532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:23.888695 15535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:23.888722 15527 server_base.cc:1034] running on GCE node
W20260620 18:39:23.888695 15533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:23.889272 15527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:23.889549 15527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:23.890715 15527 hybrid_clock.cc:648] HybridClock initialized: now 1781980763890702 us; error 41 us; skew 500 ppm
Jun 20 18:39:23 dist-test-slave-wp6g krb5kdc[15288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980763, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:23.893328 15527 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:39:23.893728 15527 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:23.893802 15527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:23.893983 15527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:23.895172 15527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "f3429a61d45a48caa42ad2edf070a410"
format_stamp: "Formatted at 2026-06-20 18:39:23 on dist-test-slave-wp6g"
server_key: "cbd9e2906234768bbfc4de4468f86927"
server_key_iv: "fce2bcdb7705c07e6161ed929b256cd0"
server_key_version: "encryptionkey@0"
I20260620 18:39:23.895546 15527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "f3429a61d45a48caa42ad2edf070a410"
format_stamp: "Formatted at 2026-06-20 18:39:23 on dist-test-slave-wp6g"
server_key: "cbd9e2906234768bbfc4de4468f86927"
server_key_iv: "fce2bcdb7705c07e6161ed929b256cd0"
server_key_version: "encryptionkey@0"
I20260620 18:39:23.896883 15527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:39:23.897643 15541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:23.897825 15527 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:23.897946 15527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "f3429a61d45a48caa42ad2edf070a410"
format_stamp: "Formatted at 2026-06-20 18:39:23 on dist-test-slave-wp6g"
server_key: "cbd9e2906234768bbfc4de4468f86927"
server_key_iv: "fce2bcdb7705c07e6161ed929b256cd0"
server_key_version: "encryptionkey@0"
I20260620 18:39:23.898030 15527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:23.905359 15527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:23.907313 15527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:23.907490 15527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:23.907728 15527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:23.908061 15527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:23.908118 15527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:23.908159 15527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:23.908186 15527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:23.914111 15527 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:36777
I20260620 18:39:23.914146 15654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:36777 every 8 connection(s)
I20260620 18:39:23.914430 15527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:39:23 dist-test-slave-wp6g krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980763, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:23.923244 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 15527
I20260620 18:39:23.923370 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:23.923410 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:23.915186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:53005 (local address 127.10.250.126:41313)
0620 18:39:23.915235 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:23.915237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:23.915422 (+ 185us) server_negotiation.cc:407] Connection header received
0620 18:39:23.915938 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:23.915939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:23.915952 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:23.916010 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:23.916948 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.917453 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.918448 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.918718 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.918804 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:23.920825 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:23.920827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:23.920827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:23.920834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:23.921990 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:23.922405 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:23.922406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:23.922407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:23.922468 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:23.922728 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:23.922729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:23.922729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:23.922860 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:23.922939 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:23.923235 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 18:39:23.923322 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:39:23.923578 11241 external_mini_cluster.cc:1428] Setting key e1f3c8ba481e5ca195eef46e42d2430d
I20260620 18:39:23.923600 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:23.915151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41313 (local address 127.10.250.66:53005)
0620 18:39:23.915293 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:23.915340 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:23.915836 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:23.916079 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:23.916083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:23.916365 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:23.916852 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:23.916857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.917546 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.917547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:23.918355 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:23.918358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.918812 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.918813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:23.918942 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:23.919398 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:23.919406 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:23.920721 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:23.922071 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:23.922077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:23.922080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:23.922296 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:23.922554 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:23.922555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:23.922556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:23.922659 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:23.922929 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:23.922935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:23.923101 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 18:39:23.923202 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:39:23.923364 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
I20260620 18:39:23.924118 15655 heartbeater.cc:347] Connected to a master server at 127.10.250.126:41313
I20260620 18:39:23.924235 15655 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:23.924393 15655 heartbeater.cc:511] Master 127.10.250.126:41313 requested a full tablet report, sending...
I20260620 18:39:23.924783 15331 ts_manager.cc:194] Registered new tserver with Master: f3429a61d45a48caa42ad2edf070a410 (127.10.250.66:36777)
I20260620 18:39:23.925174 15331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:53005
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:39:23.949529 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:41313
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:35471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:24.031339 15660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:24.031558 15660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:24.031581 15660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:24.032696 15660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:24.032735 15660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:24.032763 15660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:24.034401 15660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:35471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:41313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:24.034669 15660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:24.034861 15660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:24.040043 15668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:24.040055 15666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:24.040043 15665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:24.040812 15660 server_base.cc:1034] running on GCE node
I20260620 18:39:24.041260 15660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:24.041554 15660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:24.042733 15660 hybrid_clock.cc:648] HybridClock initialized: now 1781980764042716 us; error 38 us; skew 500 ppm
Jun 20 18:39:24 dist-test-slave-wp6g krb5kdc[15288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980764, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:24.045310 15660 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:39:24.045724 15660 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:24.045804 15660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:24.045943 15660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:24.047147 15660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "4f87398669604130898055443fe69185"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-wp6g"
server_key: "2b8e614f48c2d643b0d0ac4b3407a204"
server_key_iv: "49e2c724580f65af37a8870ba7a2d4e7"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.047514 15660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "4f87398669604130898055443fe69185"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-wp6g"
server_key: "2b8e614f48c2d643b0d0ac4b3407a204"
server_key_iv: "49e2c724580f65af37a8870ba7a2d4e7"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.048836 15660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:39:24.049582 15674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:24.049773 15660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:24.049875 15660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "4f87398669604130898055443fe69185"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-wp6g"
server_key: "2b8e614f48c2d643b0d0ac4b3407a204"
server_key_iv: "49e2c724580f65af37a8870ba7a2d4e7"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.049957 15660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:24.071056 15660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:24.073134 15660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:24.073309 15660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:24.073555 15660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:24.073899 15660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:24.073953 15660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:24.073995 15660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:24.074023 15660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:24.080503 15660 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:38515
I20260620 18:39:24.080533 15787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:38515 every 8 connection(s)
I20260620 18:39:24.080825 15660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:24.084682 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 15660
I20260620 18:39:24.084779 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:24.084985 11241 external_mini_cluster.cc:1428] Setting key 01a44b6562e8fc699afa86611e2d882e
Jun 20 18:39:24 dist-test-slave-wp6g krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980764, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:24.091583 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.081592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:46733 (local address 127.10.250.126:41313)
0620 18:39:24.081663 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.081665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.081906 (+ 241us) server_negotiation.cc:407] Connection header received
0620 18:39:24.082492 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.082493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.082508 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.082592 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.083631 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.084294 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.085779 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.086078 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.086133 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.088502 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.088504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.088505 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:24.088513 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.090094 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:24.090589 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:24.090590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:24.090591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:24.090649 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:24.090906 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:24.090906 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:24.090907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:24.091017 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:24.091071 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.091433 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.091507 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:39:24.091799 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.081580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41313 (local address 127.10.250.67:46733)
0620 18:39:24.081778 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:24.081816 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:24.082323 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:24.082690 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:24.082694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:24.082981 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:24.083506 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:24.083512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.084392 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.084393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:24.085681 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:24.085685 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.086180 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.086182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:24.086312 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.086859 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:24.086868 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:24.088394 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:24.090196 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:24.090203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:24.090205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:24.090454 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:24.090715 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:24.090716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:24.090716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:24.090846 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:24.091146 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:24.091152 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:24.091295 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 18:39:24.091401 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:39:24.091537 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":100,"threads_started":1}
I20260620 18:39:24.092280 15788 heartbeater.cc:347] Connected to a master server at 127.10.250.126:41313
I20260620 18:39:24.092368 15788 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:24.092487 15788 heartbeater.cc:511] Master 127.10.250.126:41313 requested a full tablet report, sending...
I20260620 18:39:24.092810 15331 ts_manager.cc:194] Registered new tserver with Master: 4f87398669604130898055443fe69185 (127.10.250.67:38515)
I20260620 18:39:24.093186 15331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:46733
I20260620 18:39:24.096873 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:39:24.104429 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.098162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.10.250.126:41313)
0620 18:39:24.098272 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.098274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.098289 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:39:24.098387 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.098388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.098396 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.098445 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.099050 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.099563 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.100269 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.100450 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.100504 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.101823 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.101825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.101825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:24.101833 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.103373 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:24.103703 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:24.103705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:24.103705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:24.103767 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:24.103941 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:24.103941 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:24.103942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:24.104092 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:24.104168 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.104257 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.104324 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:39:24.105983 15331 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:24.107671 15331 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:24.114413 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.110069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:39275 (local address 127.0.0.1:50556)
0620 18:39:24.110448 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:24.110457 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:24.110549 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:24.110852 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:24.110854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:24.110856 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:24.110960 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:24.110962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.111869 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.111870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:24.113116 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:24.113118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.114095 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.114096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:24.114176 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.114192 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 18:39:24.114272 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 18:39:24.114327 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":25,"thread_start_us":66,"threads_started":1}
I20260620 18:39:24.114547 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.110439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.10.250.66:36777)
0620 18:39:24.110639 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.110643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.110805 (+ 162us) server_negotiation.cc:407] Connection header received
0620 18:39:24.110874 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.110876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.110890 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.110960 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:24.111056 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.112155 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.113407 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.114149 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.114216 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.114268 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.114409 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.114449 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":103,"threads_started":1}
I20260620 18:39:24.114787 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.110269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.10.250.65:39275)
0620 18:39:24.110463 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.110469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.110588 (+ 119us) server_negotiation.cc:407] Connection header received
0620 18:39:24.110637 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.110640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.110658 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.110744 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:24.111040 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.111763 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.113247 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.114000 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.114081 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.114154 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.114657 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.114703 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
I20260620 18:39:24.114959 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.110334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:36777 (local address 127.0.0.1:51850)
0620 18:39:24.110692 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:24.110702 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:24.110750 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:24.110970 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:24.110971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:24.110973 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:24.111008 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:24.111009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.112274 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.112275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:24.113319 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:24.113321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.114225 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.114227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:24.114294 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.114302 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:39:24.114390 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:39:24.114437 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260620 18:39:24.115345 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.110069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:38515 (local address 127.0.0.1:34592)
0620 18:39:24.110252 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:24.110295 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:24.110361 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:24.110848 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:24.110851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:24.110854 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:24.110957 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:24.110960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.113386 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.113386 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:24.114039 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:24.114040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.115106 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.115107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:24.115182 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.115185 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:24.115234 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 18:39:24.115272 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":4,"thread_start_us":72,"threads_started":1}
I20260620 18:39:24.115576 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.110155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.10.250.67:38515)
0620 18:39:24.110578 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.110582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.110593 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:39:24.110637 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.110639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.110657 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.110749 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:24.111039 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.112155 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.114158 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.114947 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.115016 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.115072 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.115454 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.115500 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":97,"threads_started":1}
I20260620 18:39:24.115495 15456 tablet_service.cc:1467] Processing CreateTablet for tablet 224b5c927fee4c268726429cebd7e3bb (DEFAULT_TABLE table=test-table [id=9eb8d6367c284fb2ad08c0112c87933b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:24.115864 15456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224b5c927fee4c268726429cebd7e3bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:24.116202 15722 tablet_service.cc:1467] Processing CreateTablet for tablet 224b5c927fee4c268726429cebd7e3bb (DEFAULT_TABLE table=test-table [id=9eb8d6367c284fb2ad08c0112c87933b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:24.116521 15722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224b5c927fee4c268726429cebd7e3bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:24.118882 15805 tablet_bootstrap.cc:492] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96: Bootstrap starting.
I20260620 18:39:24.119206 15589 tablet_service.cc:1467] Processing CreateTablet for tablet 224b5c927fee4c268726429cebd7e3bb (DEFAULT_TABLE table=test-table [id=9eb8d6367c284fb2ad08c0112c87933b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:24.119324 15806 tablet_bootstrap.cc:492] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185: Bootstrap starting.
I20260620 18:39:24.119547 15589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224b5c927fee4c268726429cebd7e3bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:24.119555 15805 tablet_bootstrap.cc:654] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:24.120009 15806 tablet_bootstrap.cc:654] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:24.120091 15805 log.cc:824] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:24.120584 15806 log.cc:824] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:24.121397 15806 tablet_bootstrap.cc:492] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185: No bootstrap required, opened a new log
I20260620 18:39:24.121471 15806 ts_tablet_manager.cc:1397] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:39:24.121888 15805 tablet_bootstrap.cc:492] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96: No bootstrap required, opened a new log
I20260620 18:39:24.121948 15805 ts_tablet_manager.cc:1397] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:39:24.122949 15809 tablet_bootstrap.cc:492] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410: Bootstrap starting.
I20260620 18:39:24.123581 15809 tablet_bootstrap.cc:654] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:24.123548 15806 raft_consensus.cc:348] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e851f018b044dbba7bfd70878f8f96" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39275 } } peers { permanent_uuid: "f3429a61d45a48caa42ad2edf070a410" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36777 } } peers { permanent_uuid: "4f87398669604130898055443fe69185" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38515 } }
I20260620 18:39:24.123711 15806 raft_consensus.cc:374] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:24.123751 15806 raft_consensus.cc:729] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f87398669604130898055443fe69185, State: Initialized, Role: FOLLOWER
I20260620 18:39:24.123878 15806 consensus_queue.cc:260] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e851f018b044dbba7bfd70878f8f96" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39275 } } peers { permanent_uuid: "f3429a61d45a48caa42ad2edf070a410" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36777 } } peers { permanent_uuid: "4f87398669604130898055443fe69185" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38515 } }
I20260620 18:39:24.124091 15809 log.cc:824] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:24.124163 15806 ts_tablet_manager.cc:1428] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 18:39:24.124192 15788 heartbeater.cc:503] Master 127.10.250.126:41313 was elected leader, sending a full tablet report...
I20260620 18:39:24.124151 15805 raft_consensus.cc:348] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e851f018b044dbba7bfd70878f8f96" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39275 } } peers { permanent_uuid: "f3429a61d45a48caa42ad2edf070a410" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36777 } } peers { permanent_uuid: "4f87398669604130898055443fe69185" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38515 } }
I20260620 18:39:24.124310 15805 raft_consensus.cc:374] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:24.124349 15805 raft_consensus.cc:729] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e851f018b044dbba7bfd70878f8f96, State: Initialized, Role: FOLLOWER
I20260620 18:39:24.124466 15805 consensus_queue.cc:260] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e851f018b044dbba7bfd70878f8f96" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39275 } } peers { permanent_uuid: "f3429a61d45a48caa42ad2edf070a410" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36777 } } peers { permanent_uuid: "4f87398669604130898055443fe69185" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38515 } }
I20260620 18:39:24.124745 15805 ts_tablet_manager.cc:1428] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:24.124765 15522 heartbeater.cc:503] Master 127.10.250.126:41313 was elected leader, sending a full tablet report...
I20260620 18:39:24.125401 15809 tablet_bootstrap.cc:492] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410: No bootstrap required, opened a new log
I20260620 18:39:24.125478 15809 ts_tablet_manager.cc:1397] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 18:39:24.127202 15809 raft_consensus.cc:348] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e851f018b044dbba7bfd70878f8f96" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39275 } } peers { permanent_uuid: "f3429a61d45a48caa42ad2edf070a410" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36777 } } peers { permanent_uuid: "4f87398669604130898055443fe69185" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38515 } }
I20260620 18:39:24.127337 15809 raft_consensus.cc:374] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:24.127359 15809 raft_consensus.cc:729] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3429a61d45a48caa42ad2edf070a410, State: Initialized, Role: FOLLOWER
I20260620 18:39:24.127485 15809 consensus_queue.cc:260] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e851f018b044dbba7bfd70878f8f96" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39275 } } peers { permanent_uuid: "f3429a61d45a48caa42ad2edf070a410" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36777 } } peers { permanent_uuid: "4f87398669604130898055443fe69185" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38515 } }
I20260620 18:39:24.127786 15809 ts_tablet_manager.cc:1428] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 18:39:24.127812 15655 heartbeater.cc:503] Master 127.10.250.126:41313 was elected leader, sending a full tablet report...
W20260620 18:39:24.164980 15656 tablet.cc:2369] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:39:24.261538 15523 tablet.cc:2369] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:39:24.264415 15812 raft_consensus.cc:482] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:24.264539 15812 raft_consensus.cc:504] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e851f018b044dbba7bfd70878f8f96" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39275 } } peers { permanent_uuid: "f3429a61d45a48caa42ad2edf070a410" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36777 } } peers { permanent_uuid: "4f87398669604130898055443fe69185" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38515 } }
I20260620 18:39:24.264973 15812 leader_election.cc:288] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3429a61d45a48caa42ad2edf070a410 (127.10.250.66:36777), 4f87398669604130898055443fe69185 (127.10.250.67:38515)
I20260620 18:39:24.267926 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.265349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:45729 (local address 127.10.250.67:38515)
0620 18:39:24.265418 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.265420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.265429 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:24.265457 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.265457 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.265466 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.265533 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:24.265725 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.266163 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.267196 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.267571 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.267627 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.267633 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.267812 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.267858 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:39:24.267933 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.265246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:38515 (local address 127.10.250.65:45729)
0620 18:39:24.265360 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:24.265369 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:24.265408 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:24.265546 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:24.265546 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:24.265548 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:24.265647 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:24.265649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.266274 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.266275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:24.267097 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:24.267100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.267656 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.267657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:24.267730 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.267734 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:39:24.267785 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 18:39:24.267815 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":4,"thread_start_us":78,"threads_started":1}
I20260620 18:39:24.268005 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.265221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:59885 (local address 127.10.250.66:36777)
0620 18:39:24.265291 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.265293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.265304 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:39:24.265347 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.265348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.265369 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.265425 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:24.265626 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.266136 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.267200 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.267594 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.267692 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.267700 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.267892 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.267933 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:39:24.268044 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.265162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:36777 (local address 127.10.250.65:59885)
0620 18:39:24.265221 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:24.265233 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:24.265278 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:24.265453 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:24.265455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:24.265457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:24.265539 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:24.265541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.266225 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.266226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:24.267096 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:24.267099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.267707 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.267708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:24.267799 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.267803 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:39:24.267867 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 18:39:24.267907 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":4}
I20260620 18:39:24.268374 15742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224b5c927fee4c268726429cebd7e3bb" candidate_uuid: "15e851f018b044dbba7bfd70878f8f96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f87398669604130898055443fe69185" is_pre_election: true
I20260620 18:39:24.268357 15609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224b5c927fee4c268726429cebd7e3bb" candidate_uuid: "15e851f018b044dbba7bfd70878f8f96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3429a61d45a48caa42ad2edf070a410" is_pre_election: true
I20260620 18:39:24.268523 15609 raft_consensus.cc:2454] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e851f018b044dbba7bfd70878f8f96 in term 0.
I20260620 18:39:24.268523 15742 raft_consensus.cc:2454] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e851f018b044dbba7bfd70878f8f96 in term 0.
I20260620 18:39:24.268791 15409 leader_election.cc:302] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e851f018b044dbba7bfd70878f8f96, f3429a61d45a48caa42ad2edf070a410; no voters:
I20260620 18:39:24.268935 15812 raft_consensus.cc:2781] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:24.268990 15812 raft_consensus.cc:482] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:24.269037 15812 raft_consensus.cc:3037] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:24.269747 15812 raft_consensus.cc:504] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e851f018b044dbba7bfd70878f8f96" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39275 } } peers { permanent_uuid: "f3429a61d45a48caa42ad2edf070a410" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36777 } } peers { permanent_uuid: "4f87398669604130898055443fe69185" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38515 } }
I20260620 18:39:24.269887 15812 leader_election.cc:288] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [CANDIDATE]: Term 1 election: Requested vote from peers f3429a61d45a48caa42ad2edf070a410 (127.10.250.66:36777), 4f87398669604130898055443fe69185 (127.10.250.67:38515)
I20260620 18:39:24.270053 15609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224b5c927fee4c268726429cebd7e3bb" candidate_uuid: "15e851f018b044dbba7bfd70878f8f96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3429a61d45a48caa42ad2edf070a410"
I20260620 18:39:24.270128 15609 raft_consensus.cc:3037] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:24.270058 15742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224b5c927fee4c268726429cebd7e3bb" candidate_uuid: "15e851f018b044dbba7bfd70878f8f96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f87398669604130898055443fe69185"
I20260620 18:39:24.270211 15742 raft_consensus.cc:3037] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:24.271008 15609 raft_consensus.cc:2454] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e851f018b044dbba7bfd70878f8f96 in term 1.
I20260620 18:39:24.271143 15742 raft_consensus.cc:2454] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e851f018b044dbba7bfd70878f8f96 in term 1.
I20260620 18:39:24.271166 15409 leader_election.cc:302] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e851f018b044dbba7bfd70878f8f96, f3429a61d45a48caa42ad2edf070a410; no voters:
I20260620 18:39:24.271245 15812 raft_consensus.cc:2781] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:24.271425 15812 raft_consensus.cc:686] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [term 1 LEADER]: Becoming Leader. State: Replica: 15e851f018b044dbba7bfd70878f8f96, State: Running, Role: LEADER
I20260620 18:39:24.271504 15812 consensus_queue.cc:237] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e851f018b044dbba7bfd70878f8f96" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39275 } } peers { permanent_uuid: "f3429a61d45a48caa42ad2edf070a410" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36777 } } peers { permanent_uuid: "4f87398669604130898055443fe69185" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38515 } }
I20260620 18:39:24.272351 15331 catalog_manager.cc:5495] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15e851f018b044dbba7bfd70878f8f96 (127.10.250.65). New cstate: current_term: 1 leader_uuid: "15e851f018b044dbba7bfd70878f8f96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e851f018b044dbba7bfd70878f8f96" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3429a61d45a48caa42ad2edf070a410" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f87398669604130898055443fe69185" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38515 } health_report { overall_health: UNKNOWN } } }
I20260620 18:39:24.296005 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.293676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.10.250.65:39275)
0620 18:39:24.293721 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.293723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.293731 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:39:24.293797 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.293798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.293816 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.293890 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:24.294004 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.294481 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.295280 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.295476 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.295528 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.295662 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:24.295771 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:24.295831 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.295862 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.295877 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:39:24.297853 15742 raft_consensus.cc:1261] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185 [term 1 FOLLOWER]: Refusing update from remote peer 15e851f018b044dbba7bfd70878f8f96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:24.297869 15609 raft_consensus.cc:1261] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410 [term 1 FOLLOWER]: Refusing update from remote peer 15e851f018b044dbba7bfd70878f8f96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:24.298184 15815 consensus_queue.cc:1035] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3429a61d45a48caa42ad2edf070a410" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:24.298321 15812 consensus_queue.cc:1035] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f87398669604130898055443fe69185" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:24.301587 15819 mvcc.cc:205] Tried to move back new op lower bound from 7298993210561273856 to 7298993210456735744. Current Snapshot: MvccSnapshot[applied={T|T < 7298993210561273856 or (T in {7298993210561273856})}]
I20260620 18:39:24.301718 15820 mvcc.cc:205] Tried to move back new op lower bound from 7298993210561273856 to 7298993210456735744. Current Snapshot: MvccSnapshot[applied={T|T < 7298993210561273856}]
I20260620 18:39:24.303184 15330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:24.303287 15330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:24.304107 15330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52cd0660ee0e412b8dbdbd3b21e2902c: Sending DeleteTablet for 3 replicas of tablet 224b5c927fee4c268726429cebd7e3bb
I20260620 18:39:24.304457 15589 tablet_service.cc:1514] Processing DeleteTablet for tablet 224b5c927fee4c268726429cebd7e3bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:24 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:51850
I20260620 18:39:24.304462 15722 tablet_service.cc:1514] Processing DeleteTablet for tablet 224b5c927fee4c268726429cebd7e3bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:24 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:34592
I20260620 18:39:24.304688 15828 tablet_replica.cc:330] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410: stopping tablet replica
I20260620 18:39:24.304701 15456 tablet_service.cc:1514] Processing DeleteTablet for tablet 224b5c927fee4c268726429cebd7e3bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:24 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:50556
I20260620 18:39:24.304788 15828 raft_consensus.cc:2229] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:24.304903 15828 raft_consensus.cc:2258] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:24.304930 15830 tablet_replica.cc:330] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96: stopping tablet replica
I20260620 18:39:24.305104 15830 raft_consensus.cc:2229] T 224b5c927fee4c268726429cebd7e3bb P 15e851f018b044dbba7bfd70878f8f96 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:24.305184 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 15394
I20260620 18:39:24.305373 15828 ts_tablet_manager.cc:1905] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:24.306826 15829 tablet_replica.cc:330] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185: stopping tablet replica
I20260620 18:39:24.306931 15829 raft_consensus.cc:2229] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:24.307040 15829 raft_consensus.cc:2258] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:24.307220 15828 ts_tablet_manager.cc:1918] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:24.307283 15828 log.cc:1196] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/224b5c927fee4c268726429cebd7e3bb
I20260620 18:39:24.307548 15828 ts_tablet_manager.cc:1939] T 224b5c927fee4c268726429cebd7e3bb P f3429a61d45a48caa42ad2edf070a410: Deleting consensus metadata
I20260620 18:39:24.308009 15829 ts_tablet_manager.cc:1905] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:24.308086 15316 catalog_manager.cc:4841] TS f3429a61d45a48caa42ad2edf070a410 (127.10.250.66:36777): tablet 224b5c927fee4c268726429cebd7e3bb (table test-table [id=9eb8d6367c284fb2ad08c0112c87933b]) successfully deleted
I20260620 18:39:24.309926 15829 ts_tablet_manager.cc:1918] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:24.310009 15829 log.cc:1196] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/224b5c927fee4c268726429cebd7e3bb
I20260620 18:39:24.310271 15829 ts_tablet_manager.cc:1939] T 224b5c927fee4c268726429cebd7e3bb P 4f87398669604130898055443fe69185: Deleting consensus metadata
I20260620 18:39:24.310702 15318 catalog_manager.cc:4841] TS 4f87398669604130898055443fe69185 (127.10.250.67:38515): tablet 224b5c927fee4c268726429cebd7e3bb (table test-table [id=9eb8d6367c284fb2ad08c0112c87933b]) successfully deleted
W20260620 18:39:24.311822 15316 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:39275 (error 108)
I20260620 18:39:24.312335 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.312217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:39275 (local address 127.0.0.1:50574)
0620 18:39:24.312269 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:24.312304 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 18:39:24.312383 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 15527
W20260620 18:39:24.312489 15316 catalog_manager.cc:4584] TS 15e851f018b044dbba7bfd70878f8f96 (127.10.250.65:39275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 224b5c927fee4c268726429cebd7e3bb: Network error: Client connection negotiation failed: client connection to 127.10.250.65:39275: connect: Connection refused (error 111)
I20260620 18:39:24.317898 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 15660
I20260620 18:39:24.322324 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 15301
2026-06-20T18:39:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2038 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T18:39:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:24Z Disabled control of system clock
I20260620 18:39:24.352653 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44369
--webserver_interface=127.10.250.126
--webserver_port=0
--builtin_ntp_servers=127.10.250.84:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:44369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:39:24.435822 15835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:24.436056 15835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:24.436079 15835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:24.437228 15835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:24.437268 15835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:24.437281 15835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:24.437294 15835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:24.437307 15835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:24.438638 15835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:44369
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:24.438850 15835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:24.439114 15835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:24.444010 15841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:24.444021 15840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:24.444021 15843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:24.444192 15835 server_base.cc:1034] running on GCE node
I20260620 18:39:24.444631 15835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:24.444973 15835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:24.446162 15835 hybrid_clock.cc:648] HybridClock initialized: now 1781980764446141 us; error 41 us; skew 500 ppm
I20260620 18:39:24.447587 15835 webserver.cc:460] Webserver started at http://127.10.250.126:43967/ using document root <none> and password file <none>
I20260620 18:39:24.447861 15835 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:24.447934 15835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:24.448091 15835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:24.449208 15835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "209e8998e161435dbb20aebbf7a6bc48"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-wp6g"
server_key: "c2973850a45cad1898229c07e960ae28"
server_key_iv: "ad3935078a0381317a8a204c6d510c60"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.449644 15835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "209e8998e161435dbb20aebbf7a6bc48"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-wp6g"
server_key: "c2973850a45cad1898229c07e960ae28"
server_key_iv: "ad3935078a0381317a8a204c6d510c60"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.451115 15835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:39:24.451902 15849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:24.452082 15835 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:39:24.452193 15835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "209e8998e161435dbb20aebbf7a6bc48"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-wp6g"
server_key: "c2973850a45cad1898229c07e960ae28"
server_key_iv: "ad3935078a0381317a8a204c6d510c60"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.452272 15835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:24.455811 15835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:24.456090 15835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:24.456230 15835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:24.460117 15835 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:44369
I20260620 18:39:24.460163 15901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:44369 every 8 connection(s)
I20260620 18:39:24.460527 15835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:24.461215 15902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:24.464088 15902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48: Bootstrap starting.
I20260620 18:39:24.464648 15902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:24.464901 15902 log.cc:824] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:24.465557 15902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48: No bootstrap required, opened a new log
I20260620 18:39:24.466871 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 15835
I20260620 18:39:24.466982 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:24.467156 11241 external_mini_cluster.cc:1428] Setting key e8bd127a8e768732b208b62dc34a8402
I20260620 18:39:24.467408 15902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209e8998e161435dbb20aebbf7a6bc48" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44369 } }
I20260620 18:39:24.467602 15902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:24.467646 15902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209e8998e161435dbb20aebbf7a6bc48, State: Initialized, Role: FOLLOWER
I20260620 18:39:24.467787 15902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209e8998e161435dbb20aebbf7a6bc48" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44369 } }
I20260620 18:39:24.467900 15902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:24.467947 15902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:24.467999 15902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:24.468731 15902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209e8998e161435dbb20aebbf7a6bc48" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44369 } }
I20260620 18:39:24.468880 15902 leader_election.cc:302] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 209e8998e161435dbb20aebbf7a6bc48; no voters:
I20260620 18:39:24.469054 15902 leader_election.cc:288] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:24.469091 15907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:24.469233 15907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [term 1 LEADER]: Becoming Leader. State: Replica: 209e8998e161435dbb20aebbf7a6bc48, State: Running, Role: LEADER
I20260620 18:39:24.469342 15902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:24.469321 15907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209e8998e161435dbb20aebbf7a6bc48" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44369 } }
I20260620 18:39:24.469878 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.467711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.10.250.126:44369)
0620 18:39:24.467996 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.468003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.468054 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:39:24.468163 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.468168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.468202 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.468317 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.469188 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.469193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.469193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:24.469200 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.469249 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:24.469264 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:24.469583 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.469616 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.469677 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":126,"threads_started":1}
I20260620 18:39:24.469951 15908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "209e8998e161435dbb20aebbf7a6bc48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209e8998e161435dbb20aebbf7a6bc48" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44369 } } }
I20260620 18:39:24.470057 15908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:24.470213 15909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 209e8998e161435dbb20aebbf7a6bc48. Latest consensus state: current_term: 1 leader_uuid: "209e8998e161435dbb20aebbf7a6bc48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209e8998e161435dbb20aebbf7a6bc48" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44369 } } }
I20260620 18:39:24.470242 15916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:24.470263 15909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:24.470855 15916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:24.472303 15916 catalog_manager.cc:1347] Generated new cluster ID: 03421b115c4d4041952c65ce20f07e2f
I20260620 18:39:24.472347 15916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:24.484962 15916 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:24.485181 15916 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:24.490022 15916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48: Generated new TSK 0
I20260620 18:39:24.490370 15916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:39:24.496955 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44369
--builtin_ntp_servers=127.10.250.84:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:39:24.587363 15926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:24.587630 15926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:24.587672 15926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:24.588927 15926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:24.589002 15926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:24.589051 15926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:24.590548 15926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:24.590842 15926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:24.591171 15926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:24.596413 15934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:24.596402 15931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:24.596396 15932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:24.596989 15926 server_base.cc:1034] running on GCE node
I20260620 18:39:24.597220 15926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:24.597556 15926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:24.598742 15926 hybrid_clock.cc:648] HybridClock initialized: now 1781980764598727 us; error 29 us; skew 500 ppm
I20260620 18:39:24.600152 15926 webserver.cc:460] Webserver started at http://127.10.250.65:42381/ using document root <none> and password file <none>
I20260620 18:39:24.600415 15926 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:24.600472 15926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:24.600601 15926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:24.602007 15926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "630be49b7b6845b482046c32ca093689"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-wp6g"
server_key: "b164d572bff0cd83529ac7d4cb10327f"
server_key_iv: "9292bd96f093555f1653bdd435af5acc"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.602398 15926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "630be49b7b6845b482046c32ca093689"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-wp6g"
server_key: "b164d572bff0cd83529ac7d4cb10327f"
server_key_iv: "9292bd96f093555f1653bdd435af5acc"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.604063 15926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:24.604908 15940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:24.605115 15926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:39:24.605216 15926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "630be49b7b6845b482046c32ca093689"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-wp6g"
server_key: "b164d572bff0cd83529ac7d4cb10327f"
server_key_iv: "9292bd96f093555f1653bdd435af5acc"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.605312 15926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:24.609965 15926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:24.610199 15926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:24.610299 15926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:24.610457 15926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:24.610798 15926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:24.610833 15926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:24.610878 15926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:24.610903 15926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:24.616843 15926 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:37987
I20260620 18:39:24.616910 16053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:37987 every 8 connection(s)
I20260620 18:39:24.617249 15926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:24.620596 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.618076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:56369 (local address 127.10.250.126:44369)
0620 18:39:24.618132 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.618134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.618388 (+ 254us) server_negotiation.cc:407] Connection header received
0620 18:39:24.619003 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.619005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.619013 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.619071 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.620327 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.620328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.620329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:24.620334 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.620351 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:24.620356 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:24.620405 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.620505 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.620524 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 18:39:24.620837 16056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.618067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44369 (local address 127.10.250.65:56369)
0620 18:39:24.618265 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:24.618299 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:24.618891 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:24.619154 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:24.619158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:24.619167 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:24.620165 (+ 998us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:24.620165 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:24.620196 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:24.620225 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:24.620412 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:24.620417 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:24.620420 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:24.620515 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:39:24.620580 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260620 18:39:24.621382 16054 heartbeater.cc:347] Connected to a master server at 127.10.250.126:44369
I20260620 18:39:24.621484 16054 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:24.621631 16054 heartbeater.cc:511] Master 127.10.250.126:44369 requested a full tablet report, sending...
I20260620 18:39:24.622155 15866 ts_manager.cc:194] Registered new tserver with Master: 630be49b7b6845b482046c32ca093689 (127.10.250.65:37987)
I20260620 18:39:24.625988 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 15926
I20260620 18:39:24.626068 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:24.626255 11241 external_mini_cluster.cc:1428] Setting key 9b4eff5895dae7a978b0edfee13a1855
I20260620 18:39:24.627266 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44369
--builtin_ntp_servers=127.10.250.84:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:39:24.709113 16057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:24.709353 16057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:24.709393 16057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:24.710726 16057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:24.710801 16057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:24.710863 16057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:24.712342 16057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:24.712611 16057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:24.712917 16057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:24.718055 16065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:24.718080 16063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:24.718055 16062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:24.718318 16057 server_base.cc:1034] running on GCE node
I20260620 18:39:24.718601 16057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:24.718868 16057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:24.720055 16057 hybrid_clock.cc:648] HybridClock initialized: now 1781980764720041 us; error 29 us; skew 500 ppm
I20260620 18:39:24.721477 16057 webserver.cc:460] Webserver started at http://127.10.250.66:40753/ using document root <none> and password file <none>
I20260620 18:39:24.721741 16057 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:24.721829 16057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:24.722013 16057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:24.723207 16057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "2896faf9c98d42f3adfc17c5e0cc6ee1"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-wp6g"
server_key: "85dfcee9da5f1aafc3ec604455ca7f42"
server_key_iv: "5c4f5be2ce7bce711167b4fa7c8c0a33"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.723590 16057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "2896faf9c98d42f3adfc17c5e0cc6ee1"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-wp6g"
server_key: "85dfcee9da5f1aafc3ec604455ca7f42"
server_key_iv: "5c4f5be2ce7bce711167b4fa7c8c0a33"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.724980 16057 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:39:24.725762 16071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:24.725991 16057 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:24.726099 16057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "2896faf9c98d42f3adfc17c5e0cc6ee1"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-wp6g"
server_key: "85dfcee9da5f1aafc3ec604455ca7f42"
server_key_iv: "5c4f5be2ce7bce711167b4fa7c8c0a33"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.726186 16057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:24.729679 16057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:24.730000 16057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:24.730171 16057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:24.730404 16057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:24.730790 16057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:24.730846 16057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:24.730901 16057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:24.730929 16057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:24.736933 16057 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:34135
I20260620 18:39:24.736971 16184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:34135 every 8 connection(s)
I20260620 18:39:24.737329 16057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:24.740480 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.738156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:47041 (local address 127.10.250.126:44369)
0620 18:39:24.738224 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.738226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.738478 (+ 252us) server_negotiation.cc:407] Connection header received
0620 18:39:24.738988 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.738990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.738995 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.739050 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.740261 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.740263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.740263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:24.740267 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.740284 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:24.740288 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:24.740338 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.740412 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.740425 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:39:24.740687 16187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.738151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44369 (local address 127.10.250.66:47041)
0620 18:39:24.738346 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:24.738381 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:24.738888 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:24.739106 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:24.739110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:24.739118 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:24.740103 (+ 985us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:24.740103 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:24.740133 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:24.740163 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:24.740346 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:24.740349 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:24.740351 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:39:24.740403 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 18:39:24.740452 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260620 18:39:24.741245 16185 heartbeater.cc:347] Connected to a master server at 127.10.250.126:44369
I20260620 18:39:24.741322 16185 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:24.741540 16185 heartbeater.cc:511] Master 127.10.250.126:44369 requested a full tablet report, sending...
I20260620 18:39:24.741928 15866 ts_manager.cc:194] Registered new tserver with Master: 2896faf9c98d42f3adfc17c5e0cc6ee1 (127.10.250.66:34135)
I20260620 18:39:24.742054 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 16057
I20260620 18:39:24.742172 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:24.742362 11241 external_mini_cluster.cc:1428] Setting key aff5e4c3f0753085e9c64a6e7fe05568
I20260620 18:39:24.743355 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44369
--builtin_ntp_servers=127.10.250.84:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:39:24.825565 16188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:24.825815 16188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:24.825841 16188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:24.827229 16188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:24.827286 16188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:24.827316 16188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:24.828950 16188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:24.829239 16188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:24.829555 16188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:24.834933 16194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:24.834928 16196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:24.834950 16193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:24.835158 16188 server_base.cc:1034] running on GCE node
I20260620 18:39:24.835412 16188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:24.835624 16188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:24.836793 16188 hybrid_clock.cc:648] HybridClock initialized: now 1781980764836766 us; error 42 us; skew 500 ppm
I20260620 18:39:24.838236 16188 webserver.cc:460] Webserver started at http://127.10.250.67:44075/ using document root <none> and password file <none>
I20260620 18:39:24.838603 16188 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:24.838658 16188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:24.838810 16188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:24.839973 16188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "4895d949053642519736c9e02830e954"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-wp6g"
server_key: "54dd8a209da3350d57c044ffcc4dabea"
server_key_iv: "9bf433197bd6dacf294d200e8cc03b74"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.840353 16188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "4895d949053642519736c9e02830e954"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-wp6g"
server_key: "54dd8a209da3350d57c044ffcc4dabea"
server_key_iv: "9bf433197bd6dacf294d200e8cc03b74"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.841782 16188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:39:24.842687 16202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:24.842859 16188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:24.842948 16188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "4895d949053642519736c9e02830e954"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-wp6g"
server_key: "54dd8a209da3350d57c044ffcc4dabea"
server_key_iv: "9bf433197bd6dacf294d200e8cc03b74"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.843005 16188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:24.846561 16188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:24.846808 16188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:24.846926 16188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:24.847088 16188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:24.847350 16188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:24.847378 16188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:24.847400 16188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:24.847412 16188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:24.853046 16188 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:42211
I20260620 18:39:24.853125 16315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:42211 every 8 connection(s)
I20260620 18:39:24.853487 16188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:24.856913 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.854364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:55769 (local address 127.10.250.126:44369)
0620 18:39:24.854440 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.854443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.854743 (+ 300us) server_negotiation.cc:407] Connection header received
0620 18:39:24.855254 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.855255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.855260 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.855328 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.856624 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.856626 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.856626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:24.856631 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.856653 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:24.856657 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:24.856724 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.856819 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.856836 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:39:24.857139 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.854362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44369 (local address 127.10.250.67:55769)
0620 18:39:24.854586 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:24.854626 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:24.855159 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:24.855408 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:24.855412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:24.855420 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:24.856457 (+ 1037us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:24.856458 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:24.856488 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:24.856518 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:24.856722 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:24.856728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:24.856732 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:39:24.856830 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:39:24.856920 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260620 18:39:24.857728 16316 heartbeater.cc:347] Connected to a master server at 127.10.250.126:44369
I20260620 18:39:24.857836 16316 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:24.857991 16316 heartbeater.cc:511] Master 127.10.250.126:44369 requested a full tablet report, sending...
I20260620 18:39:24.858384 15866 ts_manager.cc:194] Registered new tserver with Master: 4895d949053642519736c9e02830e954 (127.10.250.67:42211)
I20260620 18:39:24.858419 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 16188
I20260620 18:39:24.858517 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:24.858745 11241 external_mini_cluster.cc:1428] Setting key 7ef7a00ab7891f277dea6ed5e66781c0
I20260620 18:39:24.859678 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:39:24.862048 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.860993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.10.250.126:44369)
0620 18:39:24.861032 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.861034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.861140 (+ 106us) server_negotiation.cc:407] Connection header received
0620 18:39:24.861204 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.861205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.861209 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.861260 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.861896 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.861897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.861897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:24.861900 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.861911 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:24.861915 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:24.861955 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.861988 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.862001 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:39:24.863607 15866 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:24.865355 15866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:24.869279 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.867607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:42211 (local address 127.0.0.1:45392)
0620 18:39:24.868094 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:24.868104 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:24.868164 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:24.868341 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:24.868341 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:24.868344 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:24.868823 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:24.868824 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:24.868841 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:24.868866 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:24.869105 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:24.869111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:24.869114 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:24.869177 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 18:39:24.869192 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":90,"threads_started":1}
I20260620 18:39:24.869463 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.867683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.10.250.66:34135)
0620 18:39:24.867863 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.867869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.867908 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:39:24.867986 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.867989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.867999 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.868102 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.869024 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.869034 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.869034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:24.869039 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.869075 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:24.869101 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:24.869323 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.869349 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.869366 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260620 18:39:24.869669 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.867908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.10.250.67:42211)
0620 18:39:24.868068 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.868072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.868223 (+ 151us) server_negotiation.cc:407] Connection header received
0620 18:39:24.868252 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.868254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.868263 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.868337 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.868952 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.868959 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.868960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:24.868965 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.868999 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:24.869026 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:24.869079 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.869504 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.869580 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260620 18:39:24.869383 16325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.867598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:34135 (local address 127.0.0.1:59014)
0620 18:39:24.867788 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:24.867857 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:24.867924 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:24.868107 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:24.868112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:24.868118 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:24.868914 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:24.868915 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:24.868922 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:24.868935 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:24.869181 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:24.869182 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:24.869183 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 18:39:24.869212 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 18:39:24.869225 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":60,"threads_started":1}
I20260620 18:39:24.870190 16324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.867814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:37987 (local address 127.0.0.1:48854)
0620 18:39:24.868735 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:24.868745 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:24.868793 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:24.868987 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:24.868988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:24.868991 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:24.869418 (+ 427us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:24.869418 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:24.869424 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:24.869435 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:24.869718 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:24.869719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:24.869719 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 18:39:24.870117 (+ 398us) client_negotiation.cc:240] Negotiation successful
0620 18:39:24.870131 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"thread_start_us":58,"threads_started":1}
I20260620 18:39:24.870337 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.868003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48854 (local address 127.10.250.65:37987)
0620 18:39:24.868189 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.868194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.868856 (+ 662us) server_negotiation.cc:407] Connection header received
0620 18:39:24.868896 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.868900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.868910 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.868987 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.869523 (+ 536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.869532 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.869533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:24.869538 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.869576 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:24.869603 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:24.869712 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.869805 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.869821 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
I20260620 18:39:24.870584 16119 tablet_service.cc:1467] Processing CreateTablet for tablet 8177cb7b92c14c57926cdd121eb9661b (DEFAULT_TABLE table=test-table [id=3ceace0fa6de4c70b47699f03b3c34c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:24.870575 16250 tablet_service.cc:1467] Processing CreateTablet for tablet 8177cb7b92c14c57926cdd121eb9661b (DEFAULT_TABLE table=test-table [id=3ceace0fa6de4c70b47699f03b3c34c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:24.870954 16250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8177cb7b92c14c57926cdd121eb9661b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:24.870965 16119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8177cb7b92c14c57926cdd121eb9661b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:24.871295 15988 tablet_service.cc:1467] Processing CreateTablet for tablet 8177cb7b92c14c57926cdd121eb9661b (DEFAULT_TABLE table=test-table [id=3ceace0fa6de4c70b47699f03b3c34c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:24.871663 15988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8177cb7b92c14c57926cdd121eb9661b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:24.873361 16330 tablet_bootstrap.cc:492] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954: Bootstrap starting.
I20260620 18:39:24.873980 16330 tablet_bootstrap.cc:654] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:24.874226 16330 log.cc:824] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:24.874348 16331 tablet_bootstrap.cc:492] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1: Bootstrap starting.
I20260620 18:39:24.875006 16331 tablet_bootstrap.cc:654] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:24.875190 16333 tablet_bootstrap.cc:492] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689: Bootstrap starting.
I20260620 18:39:24.875336 16330 tablet_bootstrap.cc:492] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954: No bootstrap required, opened a new log
I20260620 18:39:24.875314 16331 log.cc:824] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:24.875404 16330 ts_tablet_manager.cc:1397] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:39:24.875854 16333 tablet_bootstrap.cc:654] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:24.876085 16331 tablet_bootstrap.cc:492] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1: No bootstrap required, opened a new log
I20260620 18:39:24.876156 16331 ts_tablet_manager.cc:1397] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:39:24.876186 16333 log.cc:824] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:24.876932 16333 tablet_bootstrap.cc:492] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689: No bootstrap required, opened a new log
I20260620 18:39:24.876998 16333 ts_tablet_manager.cc:1397] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:39:24.878001 16330 raft_consensus.cc:348] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2896faf9c98d42f3adfc17c5e0cc6ee1" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 34135 } } peers { permanent_uuid: "630be49b7b6845b482046c32ca093689" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37987 } } peers { permanent_uuid: "4895d949053642519736c9e02830e954" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 42211 } }
I20260620 18:39:24.878201 16330 raft_consensus.cc:374] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:24.878239 16330 raft_consensus.cc:729] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4895d949053642519736c9e02830e954, State: Initialized, Role: FOLLOWER
I20260620 18:39:24.878407 16330 consensus_queue.cc:260] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2896faf9c98d42f3adfc17c5e0cc6ee1" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 34135 } } peers { permanent_uuid: "630be49b7b6845b482046c32ca093689" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37987 } } peers { permanent_uuid: "4895d949053642519736c9e02830e954" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 42211 } }
I20260620 18:39:24.878698 16330 ts_tablet_manager.cc:1428] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:24.878850 16316 heartbeater.cc:503] Master 127.10.250.126:44369 was elected leader, sending a full tablet report...
I20260620 18:39:24.878809 16331 raft_consensus.cc:348] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2896faf9c98d42f3adfc17c5e0cc6ee1" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 34135 } } peers { permanent_uuid: "630be49b7b6845b482046c32ca093689" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37987 } } peers { permanent_uuid: "4895d949053642519736c9e02830e954" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 42211 } }
I20260620 18:39:24.878957 16331 raft_consensus.cc:374] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:24.878995 16331 raft_consensus.cc:729] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2896faf9c98d42f3adfc17c5e0cc6ee1, State: Initialized, Role: FOLLOWER
I20260620 18:39:24.879101 16331 consensus_queue.cc:260] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2896faf9c98d42f3adfc17c5e0cc6ee1" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 34135 } } peers { permanent_uuid: "630be49b7b6845b482046c32ca093689" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37987 } } peers { permanent_uuid: "4895d949053642519736c9e02830e954" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 42211 } }
I20260620 18:39:24.879099 16333 raft_consensus.cc:348] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2896faf9c98d42f3adfc17c5e0cc6ee1" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 34135 } } peers { permanent_uuid: "630be49b7b6845b482046c32ca093689" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37987 } } peers { permanent_uuid: "4895d949053642519736c9e02830e954" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 42211 } }
I20260620 18:39:24.879245 16333 raft_consensus.cc:374] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:24.879279 16333 raft_consensus.cc:729] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 630be49b7b6845b482046c32ca093689, State: Initialized, Role: FOLLOWER
I20260620 18:39:24.879388 16331 ts_tablet_manager.cc:1428] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 18:39:24.879384 16333 consensus_queue.cc:260] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2896faf9c98d42f3adfc17c5e0cc6ee1" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 34135 } } peers { permanent_uuid: "630be49b7b6845b482046c32ca093689" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37987 } } peers { permanent_uuid: "4895d949053642519736c9e02830e954" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 42211 } }
I20260620 18:39:24.879395 16185 heartbeater.cc:503] Master 127.10.250.126:44369 was elected leader, sending a full tablet report...
I20260620 18:39:24.879617 16333 ts_tablet_manager.cc:1428] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:24.879740 16054 heartbeater.cc:503] Master 127.10.250.126:44369 was elected leader, sending a full tablet report...
I20260620 18:39:24.963099 16338 raft_consensus.cc:482] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:24.963274 16338 raft_consensus.cc:504] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2896faf9c98d42f3adfc17c5e0cc6ee1" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 34135 } } peers { permanent_uuid: "630be49b7b6845b482046c32ca093689" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37987 } } peers { permanent_uuid: "4895d949053642519736c9e02830e954" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 42211 } }
I20260620 18:39:24.963680 16338 leader_election.cc:288] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2896faf9c98d42f3adfc17c5e0cc6ee1 (127.10.250.66:34135), 4895d949053642519736c9e02830e954 (127.10.250.67:42211)
I20260620 18:39:24.965101 16056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.963895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:42211 (local address 127.10.250.65:58167)
0620 18:39:24.963950 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:24.963964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:24.964007 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:24.964169 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:24.964170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:24.964172 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:24.964720 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:24.964720 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:24.964729 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:24.964743 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:24.964949 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:24.964950 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:24.964951 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 18:39:24.965015 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 18:39:24.965031 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 18:39:24.965112 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.963965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:58167 (local address 127.10.250.67:42211)
0620 18:39:24.964028 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.964030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.964054 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:39:24.964088 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.964089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.964093 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.964160 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.964829 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.964830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.964831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:24.964835 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.964853 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:24.964857 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:24.964936 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.965037 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.965051 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 18:39:24.965286 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.963898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:34135 (local address 127.10.250.65:35741)
0620 18:39:24.964156 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:24.964172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:24.964246 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:24.964477 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:24.964478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:24.964480 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:24.964929 (+ 449us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:24.964930 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:24.964936 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:24.964951 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:24.965095 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:24.965095 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:24.965096 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 18:39:24.965161 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 18:39:24.965175 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":73,"thread_start_us":136,"threads_started":1}
I20260620 18:39:24.965389 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.963984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:35741 (local address 127.10.250.66:34135)
0620 18:39:24.964033 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.964035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.964239 (+ 204us) server_negotiation.cc:407] Connection header received
0620 18:39:24.964354 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.964355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.964362 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.964419 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.965016 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.965018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.965018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:24.965024 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.965039 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:24.965043 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:24.965100 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.965272 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.965289 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:39:24.965487 16270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8177cb7b92c14c57926cdd121eb9661b" candidate_uuid: "630be49b7b6845b482046c32ca093689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4895d949053642519736c9e02830e954" is_pre_election: true
I20260620 18:39:24.965649 16270 raft_consensus.cc:2454] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 630be49b7b6845b482046c32ca093689 in term 0.
I20260620 18:39:24.965660 16139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8177cb7b92c14c57926cdd121eb9661b" candidate_uuid: "630be49b7b6845b482046c32ca093689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2896faf9c98d42f3adfc17c5e0cc6ee1" is_pre_election: true
I20260620 18:39:24.965776 16139 raft_consensus.cc:2454] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 630be49b7b6845b482046c32ca093689 in term 0.
I20260620 18:39:24.965953 15944 leader_election.cc:302] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4895d949053642519736c9e02830e954, 630be49b7b6845b482046c32ca093689; no voters:
I20260620 18:39:24.966089 16338 raft_consensus.cc:2781] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:24.966176 16338 raft_consensus.cc:482] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:24.966210 16338 raft_consensus.cc:3037] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:24.967018 16338 raft_consensus.cc:504] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2896faf9c98d42f3adfc17c5e0cc6ee1" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 34135 } } peers { permanent_uuid: "630be49b7b6845b482046c32ca093689" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37987 } } peers { permanent_uuid: "4895d949053642519736c9e02830e954" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 42211 } }
I20260620 18:39:24.967172 16338 leader_election.cc:288] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [CANDIDATE]: Term 1 election: Requested vote from peers 2896faf9c98d42f3adfc17c5e0cc6ee1 (127.10.250.66:34135), 4895d949053642519736c9e02830e954 (127.10.250.67:42211)
I20260620 18:39:24.967331 16270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8177cb7b92c14c57926cdd121eb9661b" candidate_uuid: "630be49b7b6845b482046c32ca093689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4895d949053642519736c9e02830e954"
I20260620 18:39:24.967365 16139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8177cb7b92c14c57926cdd121eb9661b" candidate_uuid: "630be49b7b6845b482046c32ca093689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2896faf9c98d42f3adfc17c5e0cc6ee1"
I20260620 18:39:24.967422 16270 raft_consensus.cc:3037] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:24.967428 16139 raft_consensus.cc:3037] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:24.968107 16270 raft_consensus.cc:2454] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 630be49b7b6845b482046c32ca093689 in term 1.
I20260620 18:39:24.968168 16139 raft_consensus.cc:2454] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 630be49b7b6845b482046c32ca093689 in term 1.
I20260620 18:39:24.968314 15944 leader_election.cc:302] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4895d949053642519736c9e02830e954, 630be49b7b6845b482046c32ca093689; no voters:
I20260620 18:39:24.968403 16338 raft_consensus.cc:2781] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:24.968551 16338 raft_consensus.cc:686] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [term 1 LEADER]: Becoming Leader. State: Replica: 630be49b7b6845b482046c32ca093689, State: Running, Role: LEADER
I20260620 18:39:24.968652 16338 consensus_queue.cc:237] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2896faf9c98d42f3adfc17c5e0cc6ee1" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 34135 } } peers { permanent_uuid: "630be49b7b6845b482046c32ca093689" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37987 } } peers { permanent_uuid: "4895d949053642519736c9e02830e954" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 42211 } }
I20260620 18:39:24.969532 15866 catalog_manager.cc:5495] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 reported cstate change: term changed from 0 to 1, leader changed from <none> to 630be49b7b6845b482046c32ca093689 (127.10.250.65). New cstate: current_term: 1 leader_uuid: "630be49b7b6845b482046c32ca093689" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2896faf9c98d42f3adfc17c5e0cc6ee1" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 34135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "630be49b7b6845b482046c32ca093689" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4895d949053642519736c9e02830e954" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 42211 } health_report { overall_health: UNKNOWN } } }
W20260620 18:39:24.987973 16186 tablet.cc:2369] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:39:24.991623 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.990557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.10.250.65:37987)
0620 18:39:24.990616 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.990618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.990627 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:24.990669 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.990669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.990683 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.990746 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.991379 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.991381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.991381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:24.991385 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.991402 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:24.991407 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:24.991478 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.991527 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.991543 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 18:39:24.993533 16270 raft_consensus.cc:1261] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954 [term 1 FOLLOWER]: Refusing update from remote peer 630be49b7b6845b482046c32ca093689: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:24.993537 16139 raft_consensus.cc:1261] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1 [term 1 FOLLOWER]: Refusing update from remote peer 630be49b7b6845b482046c32ca093689: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:24.993901 16338 consensus_queue.cc:1035] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2896faf9c98d42f3adfc17c5e0cc6ee1" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:24.994051 16340 consensus_queue.cc:1035] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4895d949053642519736c9e02830e954" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 42211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:24.997586 16345 mvcc.cc:205] Tried to move back new op lower bound from 7298993213410660352 to 7298993213312303104. Current Snapshot: MvccSnapshot[applied={T|T < 7298993213410660352 or (T in {7298993213410660352})}]
I20260620 18:39:24.997730 16344 mvcc.cc:205] Tried to move back new op lower bound from 7298993213410660352 to 7298993213312303104. Current Snapshot: MvccSnapshot[applied={T|T < 7298993213410660352 or (T in {7298993213410660352})}]
I20260620 18:39:24.998881 15866 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:24.998965 15866 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:24.999866 15866 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 209e8998e161435dbb20aebbf7a6bc48: Sending DeleteTablet for 3 replicas of tablet 8177cb7b92c14c57926cdd121eb9661b
I20260620 18:39:25.000288 16119 tablet_service.cc:1514] Processing DeleteTablet for tablet 8177cb7b92c14c57926cdd121eb9661b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:24 UTC) from {username='slave'} at 127.0.0.1:59014
I20260620 18:39:25.000305 16250 tablet_service.cc:1514] Processing DeleteTablet for tablet 8177cb7b92c14c57926cdd121eb9661b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:24 UTC) from {username='slave'} at 127.0.0.1:45392
I20260620 18:39:25.000320 15988 tablet_service.cc:1514] Processing DeleteTablet for tablet 8177cb7b92c14c57926cdd121eb9661b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:24 UTC) from {username='slave'} at 127.0.0.1:48854
I20260620 18:39:25.000576 16353 tablet_replica.cc:330] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954: stopping tablet replica
I20260620 18:39:25.000597 16354 tablet_replica.cc:330] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689: stopping tablet replica
I20260620 18:39:25.000703 16353 raft_consensus.cc:2229] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:25.000710 16354 raft_consensus.cc:2229] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:25.000804 16354 raft_consensus.cc:2258] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:25.000814 16353 raft_consensus.cc:2258] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:25.000962 16352 tablet_replica.cc:330] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1: stopping tablet replica
I20260620 18:39:25.001056 16352 raft_consensus.cc:2229] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:25.001151 16352 raft_consensus.cc:2258] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:25.001345 16354 ts_tablet_manager.cc:1905] T 8177cb7b92c14c57926cdd121eb9661b P 630be49b7b6845b482046c32ca093689: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:25.001516 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 15926
I20260620 18:39:25.001631 16353 ts_tablet_manager.cc:1905] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:25.001715 16352 ts_tablet_manager.cc:1905] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:25.003597 16352 ts_tablet_manager.cc:1918] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:25.003674 16352 log.cc:1196] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/8177cb7b92c14c57926cdd121eb9661b
I20260620 18:39:25.003758 16353 ts_tablet_manager.cc:1918] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:25.003906 16353 log.cc:1196] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/8177cb7b92c14c57926cdd121eb9661b
I20260620 18:39:25.004033 16352 ts_tablet_manager.cc:1939] T 8177cb7b92c14c57926cdd121eb9661b P 2896faf9c98d42f3adfc17c5e0cc6ee1: Deleting consensus metadata
I20260620 18:39:25.004262 16353 ts_tablet_manager.cc:1939] T 8177cb7b92c14c57926cdd121eb9661b P 4895d949053642519736c9e02830e954: Deleting consensus metadata
I20260620 18:39:25.004942 15852 catalog_manager.cc:4841] TS 2896faf9c98d42f3adfc17c5e0cc6ee1 (127.10.250.66:34135): tablet 8177cb7b92c14c57926cdd121eb9661b (table test-table [id=3ceace0fa6de4c70b47699f03b3c34c6]) successfully deleted
I20260620 18:39:25.005080 15853 catalog_manager.cc:4841] TS 4895d949053642519736c9e02830e954 (127.10.250.67:42211): tablet 8177cb7b92c14c57926cdd121eb9661b (table test-table [id=3ceace0fa6de4c70b47699f03b3c34c6]) successfully deleted
W20260620 18:39:25.008141 15852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:37987 (error 108)
I20260620 18:39:25.008406 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 16057
I20260620 18:39:25.009164 16324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:25.008562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:37987 (local address 127.0.0.1:48882)
0620 18:39:25.009069 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:25.009118 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:37987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483}
W20260620 18:39:25.009964 15852 catalog_manager.cc:4584] TS 630be49b7b6845b482046c32ca093689 (127.10.250.65:37987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8177cb7b92c14c57926cdd121eb9661b: Network error: Client connection negotiation failed: client connection to 127.10.250.65:37987: connect: Connection refused (error 111)
I20260620 18:39:25.013727 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 16188
I20260620 18:39:25.018821 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 15835
2026-06-20T18:39:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (693 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 18:39:25.039259 11241 mini_oidc.cc:152] Starting JWKS server
I20260620 18:39:25.040370 11241 webserver.cc:460] Webserver started at https://127.0.0.1:46875/ using document root <none> and password file <none>
I20260620 18:39:25.040454 11241 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:39:25.040647 11241 webserver.cc:460] Webserver started at http://127.0.0.1:46109/ using document root <none> and password file <none>
2026-06-20T18:39:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:25Z Disabled control of system clock
I20260620 18:39:25.048946 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44713
--webserver_interface=127.10.250.126
--webserver_port=0
--builtin_ntp_servers=127.10.250.84:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:44713
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46875/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/testchainca.cert with env {}
W20260620 18:39:25.131352 16361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:25.131587 16361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:25.131613 16361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:25.132975 16361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:25.133020 16361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:25.133033 16361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:25.133046 16361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:25.133059 16361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:39:25.133070 16361 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:39:25.133101 16361 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46875/jwks.json
I20260620 18:39:25.134804 16361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:44713
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46875/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:25.135082 16361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:25.135329 16361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:39:25.140653 16361 server_base.cc:1034] running on GCE node
W20260620 18:39:25.140636 16366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:25.140689 16367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:25.140630 16369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:25.141026 16361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:25.141355 16361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:25.142563 16361 hybrid_clock.cc:648] HybridClock initialized: now 1781980765142543 us; error 34 us; skew 500 ppm
I20260620 18:39:25.143981 16361 webserver.cc:460] Webserver started at http://127.10.250.126:35569/ using document root <none> and password file <none>
I20260620 18:39:25.144263 16361 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:25.144309 16361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:25.144465 16361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:25.145617 16361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "f040b7a862fc4b3a880276899e499d21"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-wp6g"
server_key: "8a41f928f31c32ea082d90e00486befe"
server_key_iv: "321a4f78b56b0f0ef1d45bbf8afbb5e5"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.145990 16361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "f040b7a862fc4b3a880276899e499d21"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-wp6g"
server_key: "8a41f928f31c32ea082d90e00486befe"
server_key_iv: "321a4f78b56b0f0ef1d45bbf8afbb5e5"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.147478 16361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:39:25.148326 16375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:25.148537 16361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:25.148631 16361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "f040b7a862fc4b3a880276899e499d21"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-wp6g"
server_key: "8a41f928f31c32ea082d90e00486befe"
server_key_iv: "321a4f78b56b0f0ef1d45bbf8afbb5e5"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.148710 16361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:25.167197 16361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:25.167568 16361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:25.167692 16361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:25.171913 16361 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:44713
I20260620 18:39:25.171953 16427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:44713 every 8 connection(s)
I20260620 18:39:25.172325 16361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:25.173311 16428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:25.173528 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 16361
I20260620 18:39:25.173616 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:25.173810 11241 external_mini_cluster.cc:1428] Setting key a06bd302d93618c02207baca2eac94d4
I20260620 18:39:25.177120 16428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21: Bootstrap starting.
I20260620 18:39:25.177760 16428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:25.178095 16428 log.cc:824] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:25.178874 16428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21: No bootstrap required, opened a new log
I20260620 18:39:25.180048 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:25.174363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.10.250.126:44713)
0620 18:39:25.174610 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:25.174620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:25.174654 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:39:25.175177 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:25.175187 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:25.175279 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:25.175446 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:25.175637 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.177012 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.177895 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.178415 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.178589 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:25.179330 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:25.179334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:25.179334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:25.179556 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:25.179591 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:25.179603 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:25.179679 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:25.179787 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 18:39:25.179873 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":80,"threads_started":1}
I20260620 18:39:25.181099 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:39:25.181066 16428 raft_consensus.cc:348] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040b7a862fc4b3a880276899e499d21" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44713 } }
I20260620 18:39:25.181263 16428 raft_consensus.cc:374] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:25.181344 16428 raft_consensus.cc:729] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f040b7a862fc4b3a880276899e499d21, State: Initialized, Role: FOLLOWER
I20260620 18:39:25.181474 16428 consensus_queue.cc:260] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040b7a862fc4b3a880276899e499d21" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44713 } }
I20260620 18:39:25.181552 16428 raft_consensus.cc:388] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:25.181594 16428 raft_consensus.cc:482] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:25.181644 16428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:25.182461 16428 raft_consensus.cc:504] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040b7a862fc4b3a880276899e499d21" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44713 } }
I20260620 18:39:25.182606 16428 leader_election.cc:302] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f040b7a862fc4b3a880276899e499d21; no voters:
I20260620 18:39:25.182801 16428 leader_election.cc:288] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:25.182861 16433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:25.183063 16433 raft_consensus.cc:686] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [term 1 LEADER]: Becoming Leader. State: Replica: f040b7a862fc4b3a880276899e499d21, State: Running, Role: LEADER
I20260620 18:39:25.183089 16428 sys_catalog.cc:564] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:25.183153 16433 consensus_queue.cc:237] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040b7a862fc4b3a880276899e499d21" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44713 } }
I20260620 18:39:25.183545 16435 sys_catalog.cc:455] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f040b7a862fc4b3a880276899e499d21. Latest consensus state: current_term: 1 leader_uuid: "f040b7a862fc4b3a880276899e499d21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040b7a862fc4b3a880276899e499d21" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44713 } } }
I20260620 18:39:25.183643 16435 sys_catalog.cc:458] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:25.183537 16434 sys_catalog.cc:455] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f040b7a862fc4b3a880276899e499d21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040b7a862fc4b3a880276899e499d21" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44713 } } }
I20260620 18:39:25.183809 16434 sys_catalog.cc:458] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:25.183842 16440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:25.184327 16440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:25.185943 16440 catalog_manager.cc:1347] Generated new cluster ID: a522f59bf14a4926aff0df22f8ea8b72
I20260620 18:39:25.186007 16440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:25.199262 16440 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:25.199913 16440 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:25.202947 16440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f040b7a862fc4b3a880276899e499d21: Generated new TSK 0
I20260620 18:39:25.203199 16440 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 18:39:25.237320 16458 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 18:39:25.237493 16458 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:39:25.237756 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:25.214240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.10.250.126:44713)
0620 18:39:25.214293 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:25.214296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:25.214567 (+ 271us) server_negotiation.cc:407] Connection header received
0620 18:39:25.214620 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:25.214621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:25.214633 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:25.214707 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:39:25.214898 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.215450 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.216632 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.216837 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.216904 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:25.217146 (+ 242us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:39:25.237318 (+ 20172us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 18:39:25.237666 (+ 348us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54628: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":25}
W20260620 18:39:25.237845 16431 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54628: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 18:39:25.237980 16457 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:25.214109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44713 (local address 127.0.0.1:54628)
0620 18:39:25.214396 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:25.214410 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:25.214500 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:25.214738 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:25.214739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:25.214741 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:39:25.214817 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:25.214819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.215540 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.215541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:25.216521 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:25.216524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.216928 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.216930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:25.216997 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:25.217027 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:39:25.237485 (+ 20458us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 18:39:25.237612 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.250.126:44713: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":109,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.10.250.126:44713: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 18:39:25.238674 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 16361
2026-06-20T18:39:25Z chronyd exiting
I20260620 18:39:25.274996 11241 test_util.cc:175] -----------------------------------------------
I20260620 18:39:25.275084 11241 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980738687848-11241-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (237 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 18:39:25.275928 11241 mini_oidc.cc:152] Starting JWKS server
I20260620 18:39:25.276659 11241 webserver.cc:460] Webserver started at https://127.0.0.1:34079/ using document root <none> and password file <none>
I20260620 18:39:25.276702 11241 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:39:25.276875 11241 webserver.cc:460] Webserver started at http://127.0.0.1:38779/ using document root <none> and password file <none>
2026-06-20T18:39:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:25Z Disabled control of system clock
I20260620 18:39:25.284924 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:43315
--webserver_interface=127.10.250.126
--webserver_port=0
--builtin_ntp_servers=127.10.250.84:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:43315
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34079/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/testchainca.cert with env {}
W20260620 18:39:25.367189 16464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:25.367462 16464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:25.367509 16464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:25.368762 16464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:25.368836 16464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:25.368856 16464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:25.368878 16464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:25.368909 16464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:39:25.368940 16464 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:39:25.368968 16464 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34079/jwks.json
I20260620 18:39:25.370563 16464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:43315
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:43315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34079/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:25.370863 16464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:25.371088 16464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:25.376209 16472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:25.376199 16469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:25.376204 16470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:25.376731 16464 server_base.cc:1034] running on GCE node
I20260620 18:39:25.377014 16464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:25.377359 16464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:25.378607 16464 hybrid_clock.cc:648] HybridClock initialized: now 1781980765378580 us; error 43 us; skew 500 ppm
I20260620 18:39:25.379956 16464 webserver.cc:460] Webserver started at http://127.10.250.126:46237/ using document root <none> and password file <none>
I20260620 18:39:25.380211 16464 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:25.380283 16464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:25.380432 16464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:25.381585 16464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "613561ca6a45466fbd34aee14830e4fc"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-wp6g"
server_key: "cabe244ca6433141f64e4b6b29f1761a"
server_key_iv: "1308003221a9bfa1794ec07e3fb37537"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.381985 16464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "613561ca6a45466fbd34aee14830e4fc"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-wp6g"
server_key: "cabe244ca6433141f64e4b6b29f1761a"
server_key_iv: "1308003221a9bfa1794ec07e3fb37537"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.383522 16464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:39:25.384369 16478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:25.384544 16464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:25.384653 16464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "613561ca6a45466fbd34aee14830e4fc"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-wp6g"
server_key: "cabe244ca6433141f64e4b6b29f1761a"
server_key_iv: "1308003221a9bfa1794ec07e3fb37537"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.384732 16464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:25.398976 16464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:25.399329 16464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:25.399523 16464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:25.403589 16464 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:43315
I20260620 18:39:25.403666 16530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:43315 every 8 connection(s)
I20260620 18:39:25.404036 16464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:25.404806 16531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:25.407764 16531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc: Bootstrap starting.
I20260620 18:39:25.408337 16531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:25.408578 16531 log.cc:824] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:25.409365 16531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc: No bootstrap required, opened a new log
I20260620 18:39:25.409694 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 16464
I20260620 18:39:25.409797 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:25.409976 11241 external_mini_cluster.cc:1428] Setting key e0940e668c691b6bdc64614103db5c30
I20260620 18:39:25.412016 16531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "613561ca6a45466fbd34aee14830e4fc" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 43315 } }
I20260620 18:39:25.412182 16531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:25.412207 16531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613561ca6a45466fbd34aee14830e4fc, State: Initialized, Role: FOLLOWER
I20260620 18:39:25.412398 16531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "613561ca6a45466fbd34aee14830e4fc" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 43315 } }
I20260620 18:39:25.412495 16531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:25.412539 16531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:25.412600 16531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:25.413229 16531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "613561ca6a45466fbd34aee14830e4fc" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 43315 } }
I20260620 18:39:25.413364 16531 leader_election.cc:302] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 613561ca6a45466fbd34aee14830e4fc; no voters:
I20260620 18:39:25.413564 16531 leader_election.cc:288] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:25.413590 16536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:25.413700 16536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [term 1 LEADER]: Becoming Leader. State: Replica: 613561ca6a45466fbd34aee14830e4fc, State: Running, Role: LEADER
I20260620 18:39:25.413794 16536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "613561ca6a45466fbd34aee14830e4fc" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 43315 } }
I20260620 18:39:25.413859 16531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:25.414142 16538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "613561ca6a45466fbd34aee14830e4fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "613561ca6a45466fbd34aee14830e4fc" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 43315 } } }
I20260620 18:39:25.414351 16538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:25.414290 16537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 613561ca6a45466fbd34aee14830e4fc. Latest consensus state: current_term: 1 leader_uuid: "613561ca6a45466fbd34aee14830e4fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "613561ca6a45466fbd34aee14830e4fc" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 43315 } } }
I20260620 18:39:25.414650 16537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:25.415108 16543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:25.415679 16543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:25.417546 16543 catalog_manager.cc:1347] Generated new cluster ID: a2e0425e590b41edb28350362be9b765
I20260620 18:39:25.417606 16543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:25.418241 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:25.410806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51450 (local address 127.10.250.126:43315)
0620 18:39:25.411029 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:25.411037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:25.411068 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:39:25.412010 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:25.412019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:25.412110 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:25.412314 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:25.412720 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.414582 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.415877 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.416477 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.416628 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:25.417408 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:25.417413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:25.417413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:25.417657 (+ 244us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:25.417705 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:25.417758 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:25.417831 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:25.417958 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 18:39:25.418039 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260620 18:39:25.419373 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:39:25.433549 16543 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:25.434115 16543 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:25.439673 16543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 613561ca6a45466fbd34aee14830e4fc: Generated new TSK 0
I20260620 18:39:25.439879 16543 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 18:39:25.462872 16561 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 18:39:25.462989 16561 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:39:25.463310 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:25.448448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.10.250.126:43315)
0620 18:39:25.448501 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:25.448503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:25.448590 (+ 87us) server_negotiation.cc:407] Connection header received
0620 18:39:25.448640 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:25.448641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:25.448653 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:25.448746 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:39:25.448951 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.449779 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.450564 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.450897 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.450970 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:25.451147 (+ 177us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:39:25.463051 (+ 11904us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34079/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 18:39:25.463224 (+ 173us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51460: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34079/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 18:39:25.463379 16535 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51460: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34079/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 18:39:25.463439 16560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:25.448355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:43315 (local address 127.0.0.1:51460)
0620 18:39:25.448510 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:25.448521 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:25.448586 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:25.448800 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:25.448801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:25.448803 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:39:25.448860 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:25.448871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.449870 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.449871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:25.450466 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:25.450468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.450984 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.450985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:25.451032 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:25.451037 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:39:25.463187 (+ 12150us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34079/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 18:39:25.463298 (+ 111us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.250.126:43315: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34079/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":101,"threads_started":1}
I20260620 18:39:25.463708 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 16464
2026-06-20T18:39:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (225 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 18:39:25.501242 11241 mini_oidc.cc:152] Starting JWKS server
I20260620 18:39:25.501971 11241 webserver.cc:460] Webserver started at https://127.0.0.1:44251/ using document root <none> and password file <none>
I20260620 18:39:25.502043 11241 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:39:25.502215 11241 webserver.cc:460] Webserver started at http://127.0.0.1:36827/ using document root <none> and password file <none>
2026-06-20T18:39:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:25Z Disabled control of system clock
I20260620 18:39:25.510072 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:37301
--webserver_interface=127.10.250.126
--webserver_port=0
--builtin_ntp_servers=127.10.250.84:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:37301
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44251/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 18:39:25.594551 16567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:25.594834 16567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:25.594874 16567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:25.596171 16567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:25.596251 16567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:25.596272 16567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:25.596293 16567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:25.596323 16567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:39:25.596362 16567 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:39:25.596388 16567 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44251/jwks.json
I20260620 18:39:25.597924 16567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:37301
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:37301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44251/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:25.598229 16567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:25.598601 16567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:25.603749 16575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:25.603803 16572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:25.603760 16573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:25.603971 16567 server_base.cc:1034] running on GCE node
I20260620 18:39:25.604192 16567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:25.604444 16567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:25.605652 16567 hybrid_clock.cc:648] HybridClock initialized: now 1781980765605607 us; error 31 us; skew 500 ppm
I20260620 18:39:25.607084 16567 webserver.cc:460] Webserver started at http://127.10.250.126:44801/ using document root <none> and password file <none>
I20260620 18:39:25.607332 16567 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:25.607429 16567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:25.607580 16567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:25.608702 16567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "cdcfe985b81245bb818a60dd70beaff1"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-wp6g"
server_key: "3379a57b0f001432df478a8897bed23d"
server_key_iv: "a4fab2543ee406f2b6e3783fbd6365fd"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.609097 16567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "cdcfe985b81245bb818a60dd70beaff1"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-wp6g"
server_key: "3379a57b0f001432df478a8897bed23d"
server_key_iv: "a4fab2543ee406f2b6e3783fbd6365fd"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.610577 16567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:25.611418 16581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:25.611625 16567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:25.611737 16567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "cdcfe985b81245bb818a60dd70beaff1"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-wp6g"
server_key: "3379a57b0f001432df478a8897bed23d"
server_key_iv: "a4fab2543ee406f2b6e3783fbd6365fd"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.611819 16567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:25.636976 16567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:25.637333 16567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:25.637562 16567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:25.641362 16567 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:37301
I20260620 18:39:25.641420 16633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:37301 every 8 connection(s)
I20260620 18:39:25.641795 16567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:25.642621 16634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:25.644680 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 16567
I20260620 18:39:25.644798 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:25.644970 11241 external_mini_cluster.cc:1428] Setting key 19538f51252a3e18f56da0a2bd94f817
I20260620 18:39:25.645843 16634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1: Bootstrap starting.
I20260620 18:39:25.646593 16634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:25.646915 16634 log.cc:824] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:25.647661 16634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1: No bootstrap required, opened a new log
I20260620 18:39:25.649835 16634 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcfe985b81245bb818a60dd70beaff1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37301 } }
I20260620 18:39:25.650013 16634 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:25.650055 16634 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdcfe985b81245bb818a60dd70beaff1, State: Initialized, Role: FOLLOWER
I20260620 18:39:25.650182 16634 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcfe985b81245bb818a60dd70beaff1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37301 } }
I20260620 18:39:25.650276 16634 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:25.650312 16634 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:25.650370 16634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:25.651050 16634 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcfe985b81245bb818a60dd70beaff1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37301 } }
I20260620 18:39:25.651166 16634 leader_election.cc:302] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdcfe985b81245bb818a60dd70beaff1; no voters:
I20260620 18:39:25.651192 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:25.645500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46884 (local address 127.10.250.126:37301)
0620 18:39:25.645725 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:25.645735 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:25.645775 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:39:25.646349 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:25.646358 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:25.646439 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:25.646618 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:25.646823 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.648177 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.648984 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.649680 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.649825 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:25.650481 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:25.650486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:25.650486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:25.650700 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:25.650733 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:25.650745 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:25.650823 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:25.650921 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 18:39:25.651004 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260620 18:39:25.651352 16634 leader_election.cc:288] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:25.651443 16639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:25.651588 16639 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [term 1 LEADER]: Becoming Leader. State: Replica: cdcfe985b81245bb818a60dd70beaff1, State: Running, Role: LEADER
I20260620 18:39:25.651665 16639 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcfe985b81245bb818a60dd70beaff1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37301 } }
I20260620 18:39:25.651898 16634 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:25.651961 16639 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdcfe985b81245bb818a60dd70beaff1. Latest consensus state: current_term: 1 leader_uuid: "cdcfe985b81245bb818a60dd70beaff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcfe985b81245bb818a60dd70beaff1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37301 } } }
I20260620 18:39:25.651971 16640 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdcfe985b81245bb818a60dd70beaff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdcfe985b81245bb818a60dd70beaff1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37301 } } }
I20260620 18:39:25.652045 16639 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:25.652066 16640 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:25.652058 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:39:25.652895 16643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:25.653395 16643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:25.655642 16643 catalog_manager.cc:1347] Generated new cluster ID: 43936d6f688149378068362bc23c7552
I20260620 18:39:25.655692 16643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:25.674947 16643 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:25.675551 16643 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:25.684635 16643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdcfe985b81245bb818a60dd70beaff1: Generated new TSK 0
I20260620 18:39:25.684880 16643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:39:25.702910 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:25.698877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.10.250.126:37301)
0620 18:39:25.698948 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:25.698950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:25.699046 (+ 96us) server_negotiation.cc:407] Connection header received
0620 18:39:25.699118 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:25.699120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:25.699135 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:25.699226 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:39:25.699351 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.700147 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.700992 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.701309 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.701384 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:25.701596 (+ 212us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:39:25.702653 (+ 1057us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44251/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 18:39:25.702829 (+ 176us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44251/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 18:39:25.702981 16637 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44251/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 18:39:25.702989 16663 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:25.698777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:37301 (local address 127.0.0.1:46886)
0620 18:39:25.698969 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:25.698981 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:25.699053 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:25.699230 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:25.699231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:25.699233 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:39:25.699287 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:25.699288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.700252 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.700253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:25.700896 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:25.700898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.701417 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.701418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:25.701471 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:25.701478 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:39:25.702763 (+ 1285us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44251/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 18:39:25.702882 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.250.126:37301: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44251/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":141,"threads_started":1}
I20260620 18:39:25.703644 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 16567
E20260620 18:39:25.713791 16664 webserver.cc:533] Webserver: sslize failed:
E20260620 18:39:25.713871 16664 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T18:39:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (239 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T18:39:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:25Z Disabled control of system clock
I20260620 18:39:25.749516 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38361
--webserver_interface=127.10.250.126
--webserver_port=0
--builtin_ntp_servers=127.10.250.84:43663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:38361
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/testchainca.cert with env {}
W20260620 18:39:25.834064 16668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:25.834277 16668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:25.834303 16668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:25.835486 16668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:25.835525 16668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:25.835537 16668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:25.835551 16668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:25.835562 16668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:39:25.835574 16668 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:39:25.835587 16668 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 18:39:25.837073 16668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:43663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:38361
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:25.837268 16668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:25.837442 16668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:25.842530 16674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:25.842538 16673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:25.842504 16676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:25.842796 16668 server_base.cc:1034] running on GCE node
I20260620 18:39:25.843027 16668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:25.843288 16668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:25.844461 16668 hybrid_clock.cc:648] HybridClock initialized: now 1781980765844446 us; error 32 us; skew 500 ppm
I20260620 18:39:25.845844 16668 webserver.cc:460] Webserver started at http://127.10.250.126:35111/ using document root <none> and password file <none>
I20260620 18:39:25.846136 16668 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:25.846189 16668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:25.846359 16668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:25.847543 16668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "46da345734b1431ebd599fb683c0daf8"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-wp6g"
server_key: "5730f87cd6bbd2fb5341b11e955f82c2"
server_key_iv: "88a3c283241f03aabb071cdb3dfaea23"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.847915 16668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "46da345734b1431ebd599fb683c0daf8"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-wp6g"
server_key: "5730f87cd6bbd2fb5341b11e955f82c2"
server_key_iv: "88a3c283241f03aabb071cdb3dfaea23"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.849244 16668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:39:25.849997 16682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:25.850176 16668 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:25.850281 16668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "46da345734b1431ebd599fb683c0daf8"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-wp6g"
server_key: "5730f87cd6bbd2fb5341b11e955f82c2"
server_key_iv: "88a3c283241f03aabb071cdb3dfaea23"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.850368 16668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:25.872983 16668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:25.873351 16668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:25.873526 16668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:25.877550 16668 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:38361
I20260620 18:39:25.877586 16734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:38361 every 8 connection(s)
I20260620 18:39:25.877947 16668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:25.879035 16735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:25.881283 16735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8: Bootstrap starting.
I20260620 18:39:25.881834 16735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:25.882076 16735 log.cc:824] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:25.882804 16735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8: No bootstrap required, opened a new log
I20260620 18:39:25.884218 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 16668
I20260620 18:39:25.884322 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:25.884509 11241 external_mini_cluster.cc:1428] Setting key 7d1ad256fc91f8d1796b9b34bf75a8e8
I20260620 18:39:25.884483 16735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46da345734b1431ebd599fb683c0daf8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38361 } }
I20260620 18:39:25.884676 16735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:25.884702 16735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46da345734b1431ebd599fb683c0daf8, State: Initialized, Role: FOLLOWER
I20260620 18:39:25.884832 16735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46da345734b1431ebd599fb683c0daf8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38361 } }
I20260620 18:39:25.884919 16735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:25.884950 16735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:25.884994 16735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:25.885682 16735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46da345734b1431ebd599fb683c0daf8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38361 } }
I20260620 18:39:25.885800 16735 leader_election.cc:302] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46da345734b1431ebd599fb683c0daf8; no voters:
I20260620 18:39:25.885998 16735 leader_election.cc:288] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:25.886165 16740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:25.886262 16735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:25.886291 16740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [term 1 LEADER]: Becoming Leader. State: Replica: 46da345734b1431ebd599fb683c0daf8, State: Running, Role: LEADER
I20260620 18:39:25.886370 16740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46da345734b1431ebd599fb683c0daf8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38361 } }
I20260620 18:39:25.886720 16741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46da345734b1431ebd599fb683c0daf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46da345734b1431ebd599fb683c0daf8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38361 } } }
I20260620 18:39:25.886821 16741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:25.886720 16742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46da345734b1431ebd599fb683c0daf8. Latest consensus state: current_term: 1 leader_uuid: "46da345734b1431ebd599fb683c0daf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46da345734b1431ebd599fb683c0daf8" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38361 } } }
I20260620 18:39:25.886994 16742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:25.887030 16748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:25.887552 16748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:25.889293 16748 catalog_manager.cc:1347] Generated new cluster ID: ec3da68864ba4d6296e1009dd0fee2d1
I20260620 18:39:25.889400 16748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:25.890830 16739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:25.884994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.10.250.126:38361)
0620 18:39:25.885201 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:25.885209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:25.885292 (+ 83us) server_negotiation.cc:407] Connection header received
0620 18:39:25.885460 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:25.885466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:25.885526 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:25.885637 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:25.885876 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.887428 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.888626 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.889178 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.889361 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:25.890073 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:25.890077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:25.890078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:25.890260 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:25.890294 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:25.890305 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:25.890396 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:25.890472 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 18:39:25.890669 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260620 18:39:25.891709 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:39:25.922765 16748 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:25.923367 16748 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:25.929551 16748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46da345734b1431ebd599fb683c0daf8: Generated new TSK 0
I20260620 18:39:25.929772 16748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:39:26.112882 16739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:25.938568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35016 (local address 127.10.250.126:38361)
0620 18:39:25.938625 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:25.938628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:25.938752 (+ 124us) server_negotiation.cc:407] Connection header received
0620 18:39:25.938834 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:25.938834 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:25.938845 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:25.938915 (+ 70us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:39:25.939102 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.939598 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.940733 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.940928 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.940979 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:25.941189 (+ 210us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:39:26.112650 (+171461us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 18:39:26.112820 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35016: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 18:39:26.112957 16739 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35016: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 18:39:26.113135 16764 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:25.938471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38361 (local address 127.0.0.1:35016)
0620 18:39:25.938677 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:25.938688 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:25.938757 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:25.938927 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:25.938928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:25.938931 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:39:25.939005 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:25.939008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.939706 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.939707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:25.940636 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:25.940639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.941007 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.941008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:25.941075 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:25.941081 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:39:26.112827 (+171746us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 18:39:26.113006 (+ 179us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.250.126:38361: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":148,"threads_started":1}
I20260620 18:39:26.113495 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 16668
2026-06-20T18:39:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (390 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (90 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:26 dist-test-slave-wp6g krb5kdc[16776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:26 dist-test-slave-wp6g krb5kdc[16776](info): set up 2 sockets
Jun 20 18:39:26 dist-test-slave-wp6g krb5kdc[16776](info): commencing operation
krb5kdc: starting...
W20260620 18:39:27.330837 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:39:27 dist-test-slave-wp6g krb5kdc[16776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:39:27.458489 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:37931
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:33357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:37931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:27.541656 16789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:27.541916 16789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:27.541957 16789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:27.543191 16789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:27.543264 16789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:27.543300 16789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:27.543329 16789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:27.543358 16789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:27.544849 16789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:33357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:37931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:37931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:27.545137 16789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:27.545377 16789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:27.550472 16794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:27.550479 16795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:27.550581 16789 server_base.cc:1034] running on GCE node
W20260620 18:39:27.550482 16797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:27.550997 16789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:27.551319 16789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:27.552501 16789 hybrid_clock.cc:648] HybridClock initialized: now 1781980767552470 us; error 52 us; skew 500 ppm
Jun 20 18:39:27 dist-test-slave-wp6g krb5kdc[16776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980767, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:27.555219 16789 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:39:27.555742 16789 webserver.cc:460] Webserver started at http://127.10.250.126:42289/ using document root <none> and password file <none>
I20260620 18:39:27.555998 16789 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:27.556066 16789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:27.556213 16789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:27.557307 16789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "a8a1429cfcd347598edb4aeccee7101c"
format_stamp: "Formatted at 2026-06-20 18:39:27 on dist-test-slave-wp6g"
server_key: "aae2f15068036eb78c5cabe87140018e"
server_key_iv: "a69745bd5dbbb331bf269ff1d7fb91ef"
server_key_version: "encryptionkey@0"
I20260620 18:39:27.557673 16789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "a8a1429cfcd347598edb4aeccee7101c"
format_stamp: "Formatted at 2026-06-20 18:39:27 on dist-test-slave-wp6g"
server_key: "aae2f15068036eb78c5cabe87140018e"
server_key_iv: "a69745bd5dbbb331bf269ff1d7fb91ef"
server_key_version: "encryptionkey@0"
I20260620 18:39:27.559238 16789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:27.560175 16804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:27.560431 16789 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:27.560544 16789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "a8a1429cfcd347598edb4aeccee7101c"
format_stamp: "Formatted at 2026-06-20 18:39:27 on dist-test-slave-wp6g"
server_key: "aae2f15068036eb78c5cabe87140018e"
server_key_iv: "a69745bd5dbbb331bf269ff1d7fb91ef"
server_key_version: "encryptionkey@0"
I20260620 18:39:27.560626 16789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:27.571599 16789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:27.574230 16789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:27.574419 16789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:27.578842 16789 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:37931
I20260620 18:39:27.578886 16856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:37931 every 8 connection(s)
I20260620 18:39:27.579293 16789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:27.580231 16857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:27.582687 16857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c: Bootstrap starting.
I20260620 18:39:27.583333 16857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:27.583451 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 16789
I20260620 18:39:27.583559 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:27.583704 16857 log.cc:824] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:27.583739 11241 external_mini_cluster.cc:1428] Setting key 80c8db7a4229449da67681c25b6a2ba4
I20260620 18:39:27.584812 16857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c: No bootstrap required, opened a new log
I20260620 18:39:27.587456 16857 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a1429cfcd347598edb4aeccee7101c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37931 } }
I20260620 18:39:27.587656 16857 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:27.587694 16857 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a1429cfcd347598edb4aeccee7101c, State: Initialized, Role: FOLLOWER
I20260620 18:39:27.587827 16857 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a1429cfcd347598edb4aeccee7101c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37931 } }
I20260620 18:39:27.587908 16857 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:27.587940 16857 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:27.587978 16857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:27.588721 16857 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a1429cfcd347598edb4aeccee7101c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37931 } }
I20260620 18:39:27.588817 16857 leader_election.cc:302] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8a1429cfcd347598edb4aeccee7101c; no voters:
I20260620 18:39:27.589064 16857 leader_election.cc:288] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:27.589094 16862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:27.589221 16862 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [term 1 LEADER]: Becoming Leader. State: Replica: a8a1429cfcd347598edb4aeccee7101c, State: Running, Role: LEADER
I20260620 18:39:27.589306 16862 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a1429cfcd347598edb4aeccee7101c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37931 } }
I20260620 18:39:27.589347 16857 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:27.589893 16863 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8a1429cfcd347598edb4aeccee7101c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a1429cfcd347598edb4aeccee7101c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37931 } } }
I20260620 18:39:27.590006 16863 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:27.590215 16864 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8a1429cfcd347598edb4aeccee7101c. Latest consensus state: current_term: 1 leader_uuid: "a8a1429cfcd347598edb4aeccee7101c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a1429cfcd347598edb4aeccee7101c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37931 } } }
I20260620 18:39:27.590504 16864 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:27.590692 16871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:27.591264 16871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 18:39:27 dist-test-slave-wp6g krb5kdc[16776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:27.593256 16871 catalog_manager.cc:1347] Generated new cluster ID: 55e8777dbd6c4381bc880fb17cd32321
I20260620 18:39:27.593313 16871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:27.596343 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:27.584350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35184 (local address 127.10.250.126:37931)
0620 18:39:27.584532 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:27.584540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:27.584576 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:39:27.585129 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:27.585138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:27.585193 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:27.585350 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:27.586296 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:27.587271 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:27.588369 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:27.588911 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:27.589058 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:27.592253 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:27.592258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:27.592259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:27.592272 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:27.594601 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:27.595055 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:27.595058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:27.595059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:27.595164 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:27.595409 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:27.595410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:27.595410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:27.595706 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:27.595847 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:27.596004 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 18:39:27.596132 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260620 18:39:27.619239 16871 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:27.620059 16871 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:27.627609 16871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8a1429cfcd347598edb4aeccee7101c: Generated new TSK 0
I20260620 18:39:27.627863 16871 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:39:27.655710 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37931
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:33357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:27.738308 16883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:27.738633 16883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:27.738682 16883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:27.739970 16883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:27.740056 16883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:27.740105 16883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:27.741758 16883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:33357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:27.742087 16883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:27.742347 16883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:27.747742 16889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:27.747835 16891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:27.747802 16888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:27.748252 16883 server_base.cc:1034] running on GCE node
I20260620 18:39:27.748653 16883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:27.748929 16883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:27.750097 16883 hybrid_clock.cc:648] HybridClock initialized: now 1781980767750073 us; error 41 us; skew 500 ppm
Jun 20 18:39:27 dist-test-slave-wp6g krb5kdc[16776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980767, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:27.752952 16883 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:39:27.753458 16883 webserver.cc:460] Webserver started at http://127.10.250.65:41707/ using document root <none> and password file <none>
I20260620 18:39:27.753724 16883 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:27.753774 16883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:27.753957 16883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:27.755144 16883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "766a100170a848caa68fd1c86d88fac4"
format_stamp: "Formatted at 2026-06-20 18:39:27 on dist-test-slave-wp6g"
server_key: "d4380080cc49688d6a7bfea127e24fbd"
server_key_iv: "ddbc3a956577635c859d15fd28ee42e6"
server_key_version: "encryptionkey@0"
I20260620 18:39:27.755532 16883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "766a100170a848caa68fd1c86d88fac4"
format_stamp: "Formatted at 2026-06-20 18:39:27 on dist-test-slave-wp6g"
server_key: "d4380080cc49688d6a7bfea127e24fbd"
server_key_iv: "ddbc3a956577635c859d15fd28ee42e6"
server_key_version: "encryptionkey@0"
I20260620 18:39:27.756942 16883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:39:27.757687 16898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:27.757912 16883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:27.758000 16883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "766a100170a848caa68fd1c86d88fac4"
format_stamp: "Formatted at 2026-06-20 18:39:27 on dist-test-slave-wp6g"
server_key: "d4380080cc49688d6a7bfea127e24fbd"
server_key_iv: "ddbc3a956577635c859d15fd28ee42e6"
server_key_version: "encryptionkey@0"
I20260620 18:39:27.758059 16883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:27.779781 16883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:27.782222 16883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:27.782346 16883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:27.782639 16883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:27.782989 16883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:27.783022 16883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:27.783067 16883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:27.783093 16883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:27.789031 16883 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:37255
I20260620 18:39:27.789076 17011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:37255 every 8 connection(s)
I20260620 18:39:27.789467 16883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:27.791378 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 16883
I20260620 18:39:27.791497 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:27.791678 11241 external_mini_cluster.cc:1428] Setting key fe122aaae66342a74051d48b0dc86597
Jun 20 18:39:27 dist-test-slave-wp6g krb5kdc[16776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980767, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
I20260620 18:39:27.800685 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:27.790233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:37861 (local address 127.10.250.126:37931)
0620 18:39:27.790274 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:27.790277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:27.790501 (+ 224us) server_negotiation.cc:407] Connection header received
0620 18:39:27.791792 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:27.791794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:27.791811 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:27.791903 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:27.792588 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:27.793105 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:27.794450 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:27.794732 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:27.794827 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:27.797431 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:27.797433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:27.797433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:27.797444 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:27.799007 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:27.799501 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:27.799503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:27.799503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:27.799570 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:27.799844 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:27.799845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:27.799845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:27.799974 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:27.800058 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:27.800501 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 18:39:27.800591 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 18:39:27.800710 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:27.790194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:37931 (local address 127.10.250.65:37861)
0620 18:39:27.790355 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:27.790395 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:27.791590 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:27.791984 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:27.791988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:27.792253 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:27.792490 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:27.792497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:27.793241 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:27.793242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:27.794349 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:27.794353 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:27.794838 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:27.794839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:27.795005 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:27.795571 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:27.795582 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:27.797307 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:27.799137 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:27.799145 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:27.799148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:27.799404 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:27.799639 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:27.799640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:27.799640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:27.799768 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:27.800069 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:27.800078 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:27.800270 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 18:39:27.800389 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 18:39:27.800531 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260620 18:39:27.801488 17012 heartbeater.cc:347] Connected to a master server at 127.10.250.126:37931
I20260620 18:39:27.801626 17012 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:27.801821 17012 heartbeater.cc:511] Master 127.10.250.126:37931 requested a full tablet report, sending...
I20260620 18:39:27.802410 16821 ts_manager.cc:194] Registered new tserver with Master: 766a100170a848caa68fd1c86d88fac4 (127.10.250.65:37255)
I20260620 18:39:27.803095 16821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:37861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:39:27.819581 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37931
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:33357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:27.904203 17017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:27.904428 17017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:27.904454 17017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:27.905704 17017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:27.905753 17017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:27.905783 17017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:27.907436 17017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:33357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:27.907712 17017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:27.907967 17017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:27.913290 17022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:27.913290 17023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:27.913355 17017 server_base.cc:1034] running on GCE node
W20260620 18:39:27.913327 17025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:27.913815 17017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:27.914057 17017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:27.915278 17017 hybrid_clock.cc:648] HybridClock initialized: now 1781980767915259 us; error 26 us; skew 500 ppm
Jun 20 18:39:27 dist-test-slave-wp6g krb5kdc[16776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980767, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:27.918066 17017 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:39:27.918661 17017 webserver.cc:460] Webserver started at http://127.10.250.66:41967/ using document root <none> and password file <none>
I20260620 18:39:27.918951 17017 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:27.919023 17017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:27.919165 17017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:27.920267 17017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "50a54417401f4b32bdb2eb47144fb397"
format_stamp: "Formatted at 2026-06-20 18:39:27 on dist-test-slave-wp6g"
server_key: "c108666b9e7a4183950efa0e6a8885ed"
server_key_iv: "fc7ca0559d29fa1c4fb4c7be80dbfc7e"
server_key_version: "encryptionkey@0"
I20260620 18:39:27.920653 17017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "50a54417401f4b32bdb2eb47144fb397"
format_stamp: "Formatted at 2026-06-20 18:39:27 on dist-test-slave-wp6g"
server_key: "c108666b9e7a4183950efa0e6a8885ed"
server_key_iv: "fc7ca0559d29fa1c4fb4c7be80dbfc7e"
server_key_version: "encryptionkey@0"
I20260620 18:39:27.922142 17017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:27.923098 17032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:27.923313 17017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:39:27.923429 17017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "50a54417401f4b32bdb2eb47144fb397"
format_stamp: "Formatted at 2026-06-20 18:39:27 on dist-test-slave-wp6g"
server_key: "c108666b9e7a4183950efa0e6a8885ed"
server_key_iv: "fc7ca0559d29fa1c4fb4c7be80dbfc7e"
server_key_version: "encryptionkey@0"
I20260620 18:39:27.923511 17017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:27.936128 17017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:27.938387 17017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:27.938607 17017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:27.938915 17017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:27.939257 17017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:27.939354 17017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:27.939419 17017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:27.939445 17017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:27.945889 17017 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:39573
I20260620 18:39:27.945927 17145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:39573 every 8 connection(s)
I20260620 18:39:27.946269 17017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:39:27 dist-test-slave-wp6g krb5kdc[16776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980767, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:27.955399 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 17017
I20260620 18:39:27.955518 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:27.955680 11241 external_mini_cluster.cc:1428] Setting key eb224c41b4506ba9bf24d02440a2afc7
I20260620 18:39:27.956864 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:27.947091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:51023 (local address 127.10.250.126:37931)
0620 18:39:27.947172 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:27.947174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:27.947394 (+ 220us) server_negotiation.cc:407] Connection header received
0620 18:39:27.948397 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:27.948398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:27.948409 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:27.948477 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:27.949191 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:27.949748 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:27.950806 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:27.951010 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:27.951086 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:27.953470 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:27.953471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:27.953472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:27.953480 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:27.955124 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:27.955632 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:27.955633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:27.955634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:27.955694 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:27.956053 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:27.956054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:27.956055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:27.956206 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:27.956365 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:27.956699 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 18:39:27.956778 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:39:27.957170 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:27.947077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:37931 (local address 127.10.250.66:51023)
0620 18:39:27.947249 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:27.947295 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:27.948218 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:27.948561 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:27.948564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:27.948864 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:27.949068 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:27.949073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:27.949841 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:27.949842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:27.950712 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:27.950714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:27.951120 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:27.951121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:27.951246 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:27.951876 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:27.951890 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:27.953366 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:27.955221 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:27.955229 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:27.955231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:27.955475 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:27.955773 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:27.955774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:27.955774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:27.955932 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:27.956371 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:27.956378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:27.956543 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 18:39:27.956668 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 18:39:27.956883 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":101,"threads_started":1}
I20260620 18:39:27.957708 17146 heartbeater.cc:347] Connected to a master server at 127.10.250.126:37931
I20260620 18:39:27.957839 17146 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:27.958034 17146 heartbeater.cc:511] Master 127.10.250.126:37931 requested a full tablet report, sending...
I20260620 18:39:27.958495 16821 ts_manager.cc:194] Registered new tserver with Master: 50a54417401f4b32bdb2eb47144fb397 (127.10.250.66:39573)
I20260620 18:39:27.959077 16821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:51023
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:39:27.985194 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37931
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:33357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:28.068452 17151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:28.068658 17151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:28.068689 17151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:28.069793 17151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:28.069826 17151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:28.069851 17151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:28.071259 17151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:33357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:28.071453 17151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:28.071630 17151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:39:28.076822 17151 server_base.cc:1034] running on GCE node
W20260620 18:39:28.076805 17159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:28.076822 17157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:28.076807 17156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:28.077325 17151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:28.077595 17151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:28.078768 17151 hybrid_clock.cc:648] HybridClock initialized: now 1781980768078763 us; error 40 us; skew 500 ppm
Jun 20 18:39:28 dist-test-slave-wp6g krb5kdc[16776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980768, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:28.081463 17151 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:39:28.081964 17151 webserver.cc:460] Webserver started at http://127.10.250.67:35611/ using document root <none> and password file <none>
I20260620 18:39:28.082211 17151 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:28.082258 17151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:28.082422 17151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:28.083576 17151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "cc0b32af30d143ff9a75a8c5932d61c6"
format_stamp: "Formatted at 2026-06-20 18:39:28 on dist-test-slave-wp6g"
server_key: "57a9151c92cbfe2fddd7baedf49b3155"
server_key_iv: "f89a81221824920e8bd0dfe754b209d7"
server_key_version: "encryptionkey@0"
I20260620 18:39:28.083933 17151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "cc0b32af30d143ff9a75a8c5932d61c6"
format_stamp: "Formatted at 2026-06-20 18:39:28 on dist-test-slave-wp6g"
server_key: "57a9151c92cbfe2fddd7baedf49b3155"
server_key_iv: "f89a81221824920e8bd0dfe754b209d7"
server_key_version: "encryptionkey@0"
I20260620 18:39:28.085414 17151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:39:28.086200 17166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:28.086385 17151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:28.086476 17151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "cc0b32af30d143ff9a75a8c5932d61c6"
format_stamp: "Formatted at 2026-06-20 18:39:28 on dist-test-slave-wp6g"
server_key: "57a9151c92cbfe2fddd7baedf49b3155"
server_key_iv: "f89a81221824920e8bd0dfe754b209d7"
server_key_version: "encryptionkey@0"
I20260620 18:39:28.086578 17151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:28.099414 17151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:28.102133 17151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:28.102264 17151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:28.102468 17151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:28.102871 17151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:28.102917 17151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:28.102968 17151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:28.102995 17151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:28.109037 17151 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:42975
I20260620 18:39:28.109112 17279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:42975 every 8 connection(s)
I20260620 18:39:28.109455 17151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:28.110548 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 17151
I20260620 18:39:28.110636 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:28.110811 11241 external_mini_cluster.cc:1428] Setting key 7d833f36b8e1d405f7fd90c7deb11b7f
Jun 20 18:39:28 dist-test-slave-wp6g krb5kdc[16776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980768, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:28.119303 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:28.110245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:59501 (local address 127.10.250.126:37931)
0620 18:39:28.110336 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:28.110338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:28.110549 (+ 211us) server_negotiation.cc:407] Connection header received
0620 18:39:28.111729 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:28.111730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:28.111742 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:28.111787 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:28.112435 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:28.112949 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:28.113993 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:28.114210 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:28.114295 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:28.116531 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:28.116532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:28.116533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:28.116541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:28.117844 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:28.118265 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:28.118266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:28.118267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:28.118331 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:28.118627 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:28.118628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:28.118629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:28.118756 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:28.118829 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:28.119128 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 18:39:28.119209 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:39:28.119544 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:28.110220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:37931 (local address 127.10.250.67:59501)
0620 18:39:28.110376 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:28.110419 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:28.111632 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:28.111861 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:28.111866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:28.112136 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:28.112340 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:28.112344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:28.113043 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:28.113044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:28.113862 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:28.113863 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:28.114305 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:28.114306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:28.114440 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:28.115054 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:28.115063 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:28.116434 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:28.117944 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:28.117950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:28.117953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:28.118181 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:28.118399 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:28.118400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:28.118401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:28.118566 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:28.118830 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:28.118836 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:28.118994 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 18:39:28.119095 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:39:28.119264 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260620 18:39:28.119992 17280 heartbeater.cc:347] Connected to a master server at 127.10.250.126:37931
I20260620 18:39:28.120082 17280 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:28.120223 17280 heartbeater.cc:511] Master 127.10.250.126:37931 requested a full tablet report, sending...
I20260620 18:39:28.120576 16821 ts_manager.cc:194] Registered new tserver with Master: cc0b32af30d143ff9a75a8c5932d61c6 (127.10.250.67:42975)
I20260620 18:39:28.120914 16821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:59501
I20260620 18:39:28.122789 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/test-admin.keytab.
Jun 20 18:39:28 dist-test-slave-wp6g krb5kdc[16776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:28.139935 11241 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 18:39:28.141810 17287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:28.140824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.10.250.65:37255)
0620 18:39:28.140987 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:28.141000 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:28.141011 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:39:28.141037 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:28.141038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:28.141052 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:28.141162 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:28.141700 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48074: BlockingRecv error: recv got EOF from 127.0.0.1:48074 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
W20260620 18:39:28.141857 11241 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:39:28 dist-test-slave-wp6g krb5kdc[16776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:28.142971 11241 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:39:28 dist-test-slave-wp6g krb5kdc[16776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:28.143863 11241 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:39:28 dist-test-slave-wp6g krb5kdc[16776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.65@KRBTEST.COM
I20260620 18:39:28.152400 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:28.144457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.10.250.65:37255)
0620 18:39:28.144519 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:28.144520 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:28.144528 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:39:28.144552 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:28.144553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:28.144562 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:28.144610 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:28.145007 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:28.145849 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:28.146739 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:28.147150 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:28.147342 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:28.149658 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:28.149665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:28.149666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:28.149673 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:28.151326 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:28.151673 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:28.151676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:28.151676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:28.151745 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:28.151925 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:28.151926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:28.151926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:28.152066 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:28.152228 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:28.152263 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 18:39:28.152326 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 18:39:28.154451 17287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:28.153850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.10.250.65:37255)
0620 18:39:28.153934 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:28.153936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:28.154014 (+ 78us) server_negotiation.cc:407] Connection header received
0620 18:39:28.154058 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:28.154058 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:28.154068 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:28.154113 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:28.154406 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48104: BlockingRecv error: recv got EOF from 127.0.0.1:48104 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 18:39:28.154608 11241 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:39:28 dist-test-slave-wp6g krb5kdc[16776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:28.155763 11241 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:39:28 dist-test-slave-wp6g krb5kdc[16776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:28.156810 11241 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:39:28 dist-test-slave-wp6g krb5kdc[16776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.65@KRBTEST.COM
I20260620 18:39:28.165221 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:28.157499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.10.250.65:37255)
0620 18:39:28.157583 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:28.157585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:28.157593 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:39:28.157615 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:28.157616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:28.157623 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:28.157683 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:28.158131 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:28.158772 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:28.159560 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:28.159787 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:28.159853 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:28.162340 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:28.162341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:28.162342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:28.162349 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:28.164052 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:28.164384 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:28.164385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:28.164385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:28.164439 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:28.164659 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:28.164661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:28.164661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:28.164779 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:28.164920 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:28.165022 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 18:39:28.165113 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 18:39:28.166934 17287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:28.166289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48116 (local address 127.10.250.65:37255)
0620 18:39:28.166344 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:28.166346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:28.166355 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:28.166401 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:28.166402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:28.166410 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:28.166452 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:28.166875 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48116: BlockingRecv error: recv got EOF from 127.0.0.1:48116 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 18:39:28.166976 11241 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980738687848-11241-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:39:28 dist-test-slave-wp6g krb5kdc[16776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:28.168023 11241 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:39:28 dist-test-slave-wp6g krb5kdc[16776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:28.169029 11241 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:39:28 dist-test-slave-wp6g krb5kdc[16776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.65@KRBTEST.COM
I20260620 18:39:28.176944 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:28.169728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.10.250.65:37255)
0620 18:39:28.169821 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:28.169823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:28.169913 (+ 90us) server_negotiation.cc:407] Connection header received
0620 18:39:28.169999 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:28.170000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:28.170016 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:28.170085 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:28.170541 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:28.171018 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:28.171747 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:28.171938 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:28.172025 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:28.174222 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:28.174224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:28.174224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:28.174232 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:28.175734 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:28.176155 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:28.176156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:28.176157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:28.176214 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:28.176425 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:28.176426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:28.176427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:28.176533 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:28.176644 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:28.176748 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 18:39:28.176833 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:39:28.177676 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 16883
I20260620 18:39:28.182487 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 17017
I20260620 18:39:28.187031 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 17151
I20260620 18:39:28.191905 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 16789
2026-06-20T18:39:28Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1897 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:28 dist-test-slave-wp6g krb5kdc[17300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:28 dist-test-slave-wp6g krb5kdc[17300](info): set up 2 sockets
Jun 20 18:39:28 dist-test-slave-wp6g krb5kdc[17300](info): commencing operation
krb5kdc: starting...
W20260620 18:39:29.226814 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:39:29 dist-test-slave-wp6g krb5kdc[17300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.126.keytab.
Entry for principal oryx/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.126.keytab.
I20260620 18:39:29.353801 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:46691
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.126.keytab
--principal=oryx/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:46691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:29.440917 17313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:29.441184 17313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:29.441219 17313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:29.442430 17313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:29.442505 17313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:29.442531 17313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:29.442596 17313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:29.442625 17313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:29.444167 17313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:46691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.126.keytab
--principal=oryx/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:46691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:29.444463 17313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:29.444713 17313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:29.450021 17321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:29.450032 17318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:29.450204 17319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:29.450266 17313 server_base.cc:1034] running on GCE node
I20260620 18:39:29.450650 17313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:29.450965 17313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:29.452147 17313 hybrid_clock.cc:648] HybridClock initialized: now 1781980769452123 us; error 30 us; skew 500 ppm
Jun 20 18:39:29 dist-test-slave-wp6g krb5kdc[17300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980769, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:29.459424 17313 init.cc:377] Logged in from keytab as oryx/127.10.250.126@KRBTEST.COM (short username oryx)
I20260620 18:39:29.459985 17313 webserver.cc:460] Webserver started at http://127.10.250.126:44623/ using document root <none> and password file <none>
I20260620 18:39:29.460232 17313 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:29.460301 17313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:29.460455 17313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:29.461565 17313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "91b96f5e5c59435abc16847a2b3db027"
format_stamp: "Formatted at 2026-06-20 18:39:29 on dist-test-slave-wp6g"
server_key: "51bdb2ab74d07cb6aa922ab9d0677dba"
server_key_iv: "1f5bd579bcc8c1d99b449c08101d559d"
server_key_version: "encryptionkey@0"
I20260620 18:39:29.461943 17313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "91b96f5e5c59435abc16847a2b3db027"
format_stamp: "Formatted at 2026-06-20 18:39:29 on dist-test-slave-wp6g"
server_key: "51bdb2ab74d07cb6aa922ab9d0677dba"
server_key_iv: "1f5bd579bcc8c1d99b449c08101d559d"
server_key_version: "encryptionkey@0"
I20260620 18:39:29.463446 17313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 18:39:29.464217 17328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:29.464381 17313 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:29.464493 17313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "91b96f5e5c59435abc16847a2b3db027"
format_stamp: "Formatted at 2026-06-20 18:39:29 on dist-test-slave-wp6g"
server_key: "51bdb2ab74d07cb6aa922ab9d0677dba"
server_key_iv: "1f5bd579bcc8c1d99b449c08101d559d"
server_key_version: "encryptionkey@0"
I20260620 18:39:29.464569 17313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:29.478495 17313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:29.481077 17313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:29.481252 17313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:29.485566 17313 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:46691
I20260620 18:39:29.485608 17380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:46691 every 8 connection(s)
I20260620 18:39:29.485983 17313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:29.487097 17381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:29.489302 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 17313
I20260620 18:39:29.489421 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:29.489609 11241 external_mini_cluster.cc:1428] Setting key 7b9798815efa569c80b80093fa4d5790
I20260620 18:39:29.489629 17381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027: Bootstrap starting.
I20260620 18:39:29.490360 17381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:29.490694 17381 log.cc:824] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:29.491570 17381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027: No bootstrap required, opened a new log
I20260620 18:39:29.493811 17381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } }
I20260620 18:39:29.493999 17381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:29.494031 17381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b96f5e5c59435abc16847a2b3db027, State: Initialized, Role: FOLLOWER
I20260620 18:39:29.494153 17381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } }
I20260620 18:39:29.494295 17381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:29.494344 17381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:29.494398 17381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:29.495066 17381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } }
I20260620 18:39:29.495193 17381 leader_election.cc:302] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91b96f5e5c59435abc16847a2b3db027; no voters:
I20260620 18:39:29.495358 17381 leader_election.cc:288] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:29.495538 17387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:29.495639 17381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:29.495689 17387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [term 1 LEADER]: Becoming Leader. State: Replica: 91b96f5e5c59435abc16847a2b3db027, State: Running, Role: LEADER
I20260620 18:39:29.495797 17387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } }
I20260620 18:39:29.496178 17388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91b96f5e5c59435abc16847a2b3db027. Latest consensus state: current_term: 1 leader_uuid: "91b96f5e5c59435abc16847a2b3db027" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } }
I20260620 18:39:29.496366 17388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:29.496591 17386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91b96f5e5c59435abc16847a2b3db027" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } }
I20260620 18:39:29.496687 17390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:29.496719 17386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:29.497253 17390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 18:39:29 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM
I20260620 18:39:29.499106 17390 catalog_manager.cc:1347] Generated new cluster ID: a2dd3d882d904832ba1e2c8e352f2c66
I20260620 18:39:29.499161 17390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:29.501922 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:29.490165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.10.250.126:46691)
0620 18:39:29.490347 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:29.490356 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:29.490386 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:39:29.491086 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:29.491097 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:29.491166 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:29.491309 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:29.492132 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:29.493080 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:29.494204 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:29.494856 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:29.494952 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:29.497981 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:29.497985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:29.497986 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:29.497998 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:29.500045 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:29.500438 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:29.500440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:29.500441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:29.500532 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:29.500697 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:29.500697 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:29.500697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:29.500968 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:29.501505 (+ 537us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:29.501579 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 18:39:29.501708 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260620 18:39:29.532907 17390 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:29.533668 17390 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:29.539237 17390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027: Generated new TSK 0
I20260620 18:39:29.539450 17390 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.65.keytab.
Entry for principal oryx/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.65.keytab.
I20260620 18:39:29.570626 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:46691
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.65.keytab
--principal=oryx/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:29.653342 17407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:29.653597 17407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:29.653633 17407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:29.654860 17407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:29.654932 17407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:29.654975 17407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:29.656466 17407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.65.keytab
--principal=oryx/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:46691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:29.656756 17407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:29.657002 17407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:29.662354 17412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:29.662357 17415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:29.662434 17407 server_base.cc:1034] running on GCE node
W20260620 18:39:29.662357 17413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:29.662942 17407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:29.663221 17407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:29.664364 17407 hybrid_clock.cc:648] HybridClock initialized: now 1781980769664342 us; error 36 us; skew 500 ppm
Jun 20 18:39:29 dist-test-slave-wp6g krb5kdc[17300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980769, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:29.667069 17407 init.cc:377] Logged in from keytab as oryx/127.10.250.65@KRBTEST.COM (short username oryx)
I20260620 18:39:29.667608 17407 webserver.cc:460] Webserver started at http://127.10.250.65:43523/ using document root <none> and password file <none>
I20260620 18:39:29.667858 17407 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:29.667938 17407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:29.668087 17407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:29.669239 17407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c"
format_stamp: "Formatted at 2026-06-20 18:39:29 on dist-test-slave-wp6g"
server_key: "454929df5abe12e7a67fb95e0b2de5be"
server_key_iv: "0623a177857738a9d040ec475b72ad6e"
server_key_version: "encryptionkey@0"
I20260620 18:39:29.669617 17407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c"
format_stamp: "Formatted at 2026-06-20 18:39:29 on dist-test-slave-wp6g"
server_key: "454929df5abe12e7a67fb95e0b2de5be"
server_key_iv: "0623a177857738a9d040ec475b72ad6e"
server_key_version: "encryptionkey@0"
I20260620 18:39:29.671188 17407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:29.672137 17422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:29.672431 17407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:29.672559 17407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c"
format_stamp: "Formatted at 2026-06-20 18:39:29 on dist-test-slave-wp6g"
server_key: "454929df5abe12e7a67fb95e0b2de5be"
server_key_iv: "0623a177857738a9d040ec475b72ad6e"
server_key_version: "encryptionkey@0"
I20260620 18:39:29.672646 17407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:29.707297 17407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:29.709730 17407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:29.709869 17407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:29.710067 17407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:29.710479 17407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:29.710584 17407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:29.710649 17407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:29.710682 17407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:29.717254 17407 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:33437
I20260620 18:39:29.717298 17535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:33437 every 8 connection(s)
I20260620 18:39:29.717666 17407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:39:29 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980769, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM
I20260620 18:39:29.726452 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 17407
I20260620 18:39:29.726620 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:29.726821 11241 external_mini_cluster.cc:1428] Setting key 6f6303f5709438cd8c5593742107cf94
I20260620 18:39:29.727692 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:29.718497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:59345 (local address 127.10.250.126:46691)
0620 18:39:29.718600 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:29.718602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:29.718831 (+ 229us) server_negotiation.cc:407] Connection header received
0620 18:39:29.719397 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:29.719399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:29.719412 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:29.719464 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:29.720495 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:29.721002 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:29.722050 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:29.722279 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:29.722368 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:29.724672 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:29.724673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:29.724673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:29.724682 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:29.726159 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:29.726654 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:29.726656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:29.726656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:29.726727 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:29.727004 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:29.727005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:29.727005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:29.727158 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:29.727221 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:29.727513 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 18:39:29.727603 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:39:29.727918 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:29.718481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.65:59345)
0620 18:39:29.718693 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:29.718731 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:29.719258 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:29.719541 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:29.719545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:29.720152 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:29.720367 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:29.720372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:29.721107 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:29.721109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:29.721954 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:29.721956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:29.722382 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:29.722383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:29.722541 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:29.723124 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:29.723133 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:29.724561 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:29.726265 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:29.726272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:29.726275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:29.726591 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:29.726783 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:29.726784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:29.726785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:29.726937 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:29.727225 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:29.727232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:29.727376 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 18:39:29.727481 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:39:29.727643 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":118,"threads_started":1}
I20260620 18:39:29.729368 17536 heartbeater.cc:347] Connected to a master server at 127.10.250.126:46691
I20260620 18:39:29.729481 17536 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:29.729689 17536 heartbeater.cc:511] Master 127.10.250.126:46691 requested a full tablet report, sending...
I20260620 18:39:29.730271 17345 ts_manager.cc:194] Registered new tserver with Master: c4ef94b55fa94b9c90a32215dfb2fa2c (127.10.250.65:33437)
I20260620 18:39:29.731019 17345 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:59345
WARNING: no policy specified for oryx/127.10.250.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.66.keytab.
Entry for principal oryx/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.66.keytab.
I20260620 18:39:29.754472 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:46691
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.66.keytab
--principal=oryx/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:29.839543 17541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:29.839797 17541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:29.839836 17541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:29.841032 17541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:29.841121 17541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:29.841166 17541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:29.842743 17541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.66.keytab
--principal=oryx/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:46691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:29.843034 17541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:29.843286 17541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:29.848332 17546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:29.848373 17549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:29.848389 17547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:29.848407 17541 server_base.cc:1034] running on GCE node
I20260620 18:39:29.848803 17541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:29.849066 17541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:29.850231 17541 hybrid_clock.cc:648] HybridClock initialized: now 1781980769850222 us; error 31 us; skew 500 ppm
Jun 20 18:39:29 dist-test-slave-wp6g krb5kdc[17300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980769, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:29.852921 17541 init.cc:377] Logged in from keytab as oryx/127.10.250.66@KRBTEST.COM (short username oryx)
I20260620 18:39:29.853411 17541 webserver.cc:460] Webserver started at http://127.10.250.66:35867/ using document root <none> and password file <none>
I20260620 18:39:29.853657 17541 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:29.853725 17541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:29.853875 17541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:29.855017 17541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "e4e832d399164153a9c69cae14100b88"
format_stamp: "Formatted at 2026-06-20 18:39:29 on dist-test-slave-wp6g"
server_key: "59826aae2fc257e1a30ba2111e0f13a4"
server_key_iv: "f46da311b6a0ebe3caa221817f069894"
server_key_version: "encryptionkey@0"
I20260620 18:39:29.855377 17541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "e4e832d399164153a9c69cae14100b88"
format_stamp: "Formatted at 2026-06-20 18:39:29 on dist-test-slave-wp6g"
server_key: "59826aae2fc257e1a30ba2111e0f13a4"
server_key_iv: "f46da311b6a0ebe3caa221817f069894"
server_key_version: "encryptionkey@0"
I20260620 18:39:29.856745 17541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:39:29.857563 17556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:29.857740 17541 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:29.857844 17541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "e4e832d399164153a9c69cae14100b88"
format_stamp: "Formatted at 2026-06-20 18:39:29 on dist-test-slave-wp6g"
server_key: "59826aae2fc257e1a30ba2111e0f13a4"
server_key_iv: "f46da311b6a0ebe3caa221817f069894"
server_key_version: "encryptionkey@0"
I20260620 18:39:29.857925 17541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:29.870965 17541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:29.873342 17541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:29.873517 17541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:29.873755 17541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:29.874086 17541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:29.874141 17541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:29.874185 17541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:29.874213 17541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:29.880260 17541 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:37457
I20260620 18:39:29.880303 17669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:37457 every 8 connection(s)
I20260620 18:39:29.880628 17541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:39:29 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980769, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM
I20260620 18:39:29.889649 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 17541
I20260620 18:39:29.889761 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:29.889953 11241 external_mini_cluster.cc:1428] Setting key 73a8408405e87dcb8921883b3425398e
I20260620 18:39:29.890314 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:29.881323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:46517 (local address 127.10.250.126:46691)
0620 18:39:29.881400 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:29.881402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:29.881614 (+ 212us) server_negotiation.cc:407] Connection header received
0620 18:39:29.882079 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:29.882080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:29.882088 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:29.882146 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:29.883090 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:29.883599 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:29.884586 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:29.884773 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:29.884826 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:29.887198 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:29.887200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:29.887200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:29.887208 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:29.888698 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:29.889149 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:29.889150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:29.889150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:29.889211 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:29.889507 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:29.889508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:29.889509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:29.889675 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:29.889766 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:29.890131 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 18:39:29.890220 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:39:29.890578 17672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:29.881313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.66:46517)
0620 18:39:29.881475 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:29.881518 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:29.881999 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:29.882203 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:29.882206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:29.882797 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:29.883004 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:29.883008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:29.883685 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:29.883685 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:29.884509 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:29.884511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:29.884848 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:29.884849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:29.884954 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:29.885484 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:29.885493 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:29.887055 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:29.888822 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:29.888829 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:29.888831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:29.889072 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:29.889307 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:29.889308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:29.889308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:29.889418 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:29.889783 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:29.889790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:29.889953 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 18:39:29.890106 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 18:39:29.890278 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260620 18:39:29.891083 17670 heartbeater.cc:347] Connected to a master server at 127.10.250.126:46691
I20260620 18:39:29.891188 17670 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:29.891396 17670 heartbeater.cc:511] Master 127.10.250.126:46691 requested a full tablet report, sending...
I20260620 18:39:29.891791 17345 ts_manager.cc:194] Registered new tserver with Master: e4e832d399164153a9c69cae14100b88 (127.10.250.66:37457)
I20260620 18:39:29.892318 17345 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:46517
WARNING: no policy specified for oryx/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.67.keytab.
Entry for principal oryx/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.67.keytab.
I20260620 18:39:29.918246 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:46691
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.67.keytab
--principal=oryx/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:30.004300 17675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:30.004542 17675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:30.004570 17675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:30.005843 17675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:30.005904 17675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:30.005936 17675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:30.007608 17675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.67.keytab
--principal=oryx/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:46691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:30.007916 17675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:30.008174 17675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:30.014197 17681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.014201 17680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.014256 17683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:30.014549 17675 server_base.cc:1034] running on GCE node
I20260620 18:39:30.014773 17675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:30.015043 17675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:30.016216 17675 hybrid_clock.cc:648] HybridClock initialized: now 1781980770016226 us; error 59 us; skew 500 ppm
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:30.019007 17675 init.cc:377] Logged in from keytab as oryx/127.10.250.67@KRBTEST.COM (short username oryx)
I20260620 18:39:30.019469 17675 webserver.cc:460] Webserver started at http://127.10.250.67:41451/ using document root <none> and password file <none>
I20260620 18:39:30.019706 17675 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:30.019773 17675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:30.019924 17675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:30.021027 17675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-wp6g"
server_key: "2d32dc1042b7fee2a68a813a8176af54"
server_key_iv: "3bb81a9626182764f842cf624326c149"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.021410 17675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-wp6g"
server_key: "2d32dc1042b7fee2a68a813a8176af54"
server_key_iv: "3bb81a9626182764f842cf624326c149"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.022930 17675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:30.023893 17690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.024246 17675 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:30.024398 17675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-wp6g"
server_key: "2d32dc1042b7fee2a68a813a8176af54"
server_key_iv: "3bb81a9626182764f842cf624326c149"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.024487 17675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:30.036037 17675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:30.038494 17675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:30.038789 17675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:30.039112 17675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:30.039564 17675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:30.039608 17675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.039633 17675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:30.039645 17675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.045703 17675 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:36941
I20260620 18:39:30.045744 17803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:36941 every 8 connection(s)
I20260620 18:39:30.046077 17675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM
I20260620 18:39:30.053573 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 17675
I20260620 18:39:30.053711 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:30.053925 11241 external_mini_cluster.cc:1428] Setting key 0718f63a689dd4c88ca0ab10ab5c857e
I20260620 18:39:30.056936 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.047143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:40627 (local address 127.10.250.126:46691)
0620 18:39:30.047207 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.047209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.047219 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:39:30.047757 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.047757 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.047769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.047828 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.048862 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.049381 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.050503 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.050804 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.050868 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.053456 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.053458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.053458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.053466 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.055479 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.055923 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.055924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.055924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.055988 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.056267 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.056267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.056268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.056390 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.056448 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.056782 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.056869 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:39:30.057050 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.046887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.67:40627)
0620 18:39:30.047044 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.047101 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.047646 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.047888 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.047891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.048491 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.048712 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.048717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.049505 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.049506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.050380 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.050383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.050928 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.050929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.051060 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.051788 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.051798 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.053322 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.055563 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.055570 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.055572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.055830 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.056058 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.056059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.056059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.056171 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.056462 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.056467 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.056602 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.056696 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.056831 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260620 18:39:30.057497 17804 heartbeater.cc:347] Connected to a master server at 127.10.250.126:46691
I20260620 18:39:30.057627 17804 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:30.057822 17804 heartbeater.cc:511] Master 127.10.250.126:46691 requested a full tablet report, sending...
I20260620 18:39:30.058182 17345 ts_manager.cc:194] Registered new tserver with Master: 5d3a1b3e7d4441c7ad394c3aa6409cc4 (127.10.250.67:36941)
I20260620 18:39:30.058604 17345 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:40627
I20260620 18:39:30.066318 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.250.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.250.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.250.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.125.keytab.
Entry for principal oryx/127.10.250.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.125.keytab.
I20260620 18:39:30.094197 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.125:35565
--webserver_interface=127.10.250.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.125.keytab
--principal=oryx/127.10.250.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:46691,127.10.250.125:35565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:30.177369 17809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:30.177634 17809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:30.177673 17809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:30.178996 17809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:30.179064 17809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:30.179087 17809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:30.179117 17809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:30.179140 17809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:30.180727 17809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:46691,127.10.250.125:35565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.125.keytab
--principal=oryx/127.10.250.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.125:35565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.250.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
W20260620 18:39:30.180883 17809 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.250.126:46691,127.10.250.125:35565'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 18:39:30.181041 17809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:30.181277 17809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:30.186853 17814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.186995 17817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.186851 17815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:30.187057 17809 server_base.cc:1034] running on GCE node
I20260620 18:39:30.187290 17809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:30.187582 17809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:30.188755 17809 hybrid_clock.cc:648] HybridClock initialized: now 1781980770188741 us; error 34 us; skew 500 ppm
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:30.191514 17809 init.cc:377] Logged in from keytab as oryx/127.10.250.125@KRBTEST.COM (short username oryx)
I20260620 18:39:30.192006 17809 webserver.cc:460] Webserver started at http://127.10.250.125:36231/ using document root <none> and password file <none>
I20260620 18:39:30.192248 17809 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:30.192317 17809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:30.192467 17809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:30.193538 17809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/data/instance:
uuid: "7d2401dd5ea84376b1824c16cdc174ae"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-wp6g"
server_key: "df19bb2ad48a2b9df666213dbb4b7bd6"
server_key_iv: "2f270384820a58657d8f6a1a94f82f9f"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.193899 17809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/wal/instance:
uuid: "7d2401dd5ea84376b1824c16cdc174ae"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-wp6g"
server_key: "df19bb2ad48a2b9df666213dbb4b7bd6"
server_key_iv: "2f270384820a58657d8f6a1a94f82f9f"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.195551 17809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 18:39:30.196399 17824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.196699 17809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:30.196808 17809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/wal
uuid: "7d2401dd5ea84376b1824c16cdc174ae"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-wp6g"
server_key: "df19bb2ad48a2b9df666213dbb4b7bd6"
server_key_iv: "2f270384820a58657d8f6a1a94f82f9f"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.196904 17809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:30.206390 17809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:30.208796 17809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:30.208957 17809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:30.213202 17809 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.125:35565
I20260620 18:39:30.213254 17876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.125:35565 every 8 connection(s)
I20260620 18:39:30.213631 17809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/data/info.pb
I20260620 18:39:30.214706 17877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:30.216266 17877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:39:30.219805 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 17809
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.125@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.250.125@KRBTEST.COM
I20260620 18:39:30.227150 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.217024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.10.250.126:46691)
0620 18:39:30.217120 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.217122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.217288 (+ 166us) server_negotiation.cc:407] Connection header received
0620 18:39:30.217767 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.217768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.217780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.217848 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.218633 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.219135 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.220646 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.220914 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.220967 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.223464 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.223465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.223465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.223474 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.225257 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.225840 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.225841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.225842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.225907 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.226253 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.226254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.226254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.226430 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.226516 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.226899 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.227038 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:39:30.227209 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.216986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:36750)
0620 18:39:30.217152 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.217195 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.217648 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.217926 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.217929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.218183 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.218484 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.218489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.219289 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.219290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.220563 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.220567 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.221002 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.221003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.221181 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.221794 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.221803 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.223324 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.225406 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.225414 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.225416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.225731 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.225985 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.225986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.225987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.226159 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.226507 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.226544 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.226680 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.226808 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.226993 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260620 18:39:30.228000 17877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:39:30.230033 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.220368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50672 (local address 127.10.250.125:35565)
0620 18:39:30.220553 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.220557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.220572 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:39:30.220593 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.220594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.220636 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.220719 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.221484 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.222182 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.223043 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.223500 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.223550 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.226501 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.226509 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.226509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.226538 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.228596 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.229071 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.229074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.229075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.229145 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.229398 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.229399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.229400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.229576 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.229771 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.229827 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.229912 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":148,"thread_start_us":116,"threads_started":1}
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.125@KRBTEST.COM for oryx/127.10.250.125@KRBTEST.COM
I20260620 18:39:30.235361 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.228344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.10.250.125:35565)
0620 18:39:30.228491 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.228493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.228499 (+ 6us) server_negotiation.cc:407] Connection header received
0620 18:39:30.228514 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.228515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.228523 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.228604 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.228904 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.229380 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.230065 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.230366 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.230440 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.232412 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.232413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.232414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.232420 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.234169 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.234551 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.234552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.234553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.234608 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.234802 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.234803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.234803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.234917 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.234992 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.235115 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.235219 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":107,"threads_started":1}
I20260620 18:39:30.235368 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.228284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:35565 (local address 127.0.0.1:50680)
0620 18:39:30.228336 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.228347 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.228388 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.228606 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.228607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.228745 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.228833 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.228835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.229471 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.229472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.229971 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.229973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.230457 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.230457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.230564 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.231016 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.231021 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.232262 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.234289 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.234290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.234291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.234452 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.234674 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.234675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.234676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.234730 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.234983 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.234984 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.235021 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.235089 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.235270 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":110}
I20260620 18:39:30.236833 17877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: Bootstrap starting.
I20260620 18:39:30.237520 17877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:30.237823 17877 log.cc:824] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:30.238721 17877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: No bootstrap required, opened a new log
I20260620 18:39:30.240916 17877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } }
I20260620 18:39:30.241098 17877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:30.241176 17877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d2401dd5ea84376b1824c16cdc174ae, State: Initialized, Role: FOLLOWER
I20260620 18:39:30.241334 17877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } }
I20260620 18:39:30.241744 17877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:30.242172 17884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } } }
I20260620 18:39:30.242276 17884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: This master's current role is: FOLLOWER
W20260620 18:39:30.243577 17895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:39:30.243639 17895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:39:30.245237 17809 master_runner.cc:428] Detected that this master 7d2401dd5ea84376b1824c16cdc174ae is joining an existing cluster
I20260620 18:39:30.245285 17809 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.250.125:35565
I20260620 18:39:30.253355 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.246280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.10.250.126:46691)
0620 18:39:30.246338 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.246341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.246469 (+ 128us) server_negotiation.cc:407] Connection header received
0620 18:39:30.246620 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.246621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.246635 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.246723 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.247267 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.248070 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.248866 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.249088 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.249175 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.250330 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.250332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.250333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.250341 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.252058 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.252432 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.252434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.252434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.252497 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.252729 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.252731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.252732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.252908 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.252988 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.253165 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.253250 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 18:39:30.253422 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.246224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:36764)
0620 18:39:30.246382 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.246394 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.246458 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.246776 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.246777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.247078 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.247179 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.247181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.248160 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.248160 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.248772 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.248773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.249202 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.249203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.249284 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.249776 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.249781 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.250224 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.252143 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.252144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.252145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.252356 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.252573 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.252574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.252574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.252638 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.252993 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.252994 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.253043 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.253122 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.253273 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":2,"thread_start_us":77,"threads_started":1}
I20260620 18:39:30.254484 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.246365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:35565 (local address 127.0.0.1:50690)
0620 18:39:30.246655 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.246666 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.246732 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.246901 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.246901 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.247109 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.247179 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.247181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.249059 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.249060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.249916 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.249918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.250412 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.250412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.250470 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.251010 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.251014 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.251408 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.253349 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.253349 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.253350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.253551 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.253762 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.253763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.253763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.253814 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.254104 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.254105 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.254153 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.254231 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.254342 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
I20260620 18:39:30.254558 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.246284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50690 (local address 127.10.250.125:35565)
0620 18:39:30.246323 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.246325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.246711 (+ 386us) server_negotiation.cc:407] Connection header received
0620 18:39:30.246813 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.246814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.246826 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.246895 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.248183 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.248968 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.250007 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.250334 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.250411 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.251487 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.251489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.251489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.251495 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.253279 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.253630 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.253631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.253632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.253681 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.253894 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.253895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.253895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.254033 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.254110 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.254260 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.254419 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":19}
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980769, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.126@KRBTEST.COM for oryx/127.10.250.125@KRBTEST.COM
I20260620 18:39:30.263695 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.256012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:35565 (local address 127.0.0.1:50700)
0620 18:39:30.256138 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.256182 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.256254 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.256400 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.256402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.256645 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.256694 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.256695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.257585 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.257585 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.258216 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.258218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.258699 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.258700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.258763 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.259243 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.259247 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.260748 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.262621 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.262630 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.262632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.262849 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.263054 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.263055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.263055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.263111 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.263381 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.263387 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.263434 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.263502 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.263605 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260620 18:39:30.263733 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.256057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.10.250.125:35565)
0620 18:39:30.256111 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.256113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.256245 (+ 132us) server_negotiation.cc:407] Connection header received
0620 18:39:30.256326 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.256327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.256338 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.256398 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.256765 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.257501 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.258299 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.258627 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.258699 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.260859 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.260861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.260861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.260868 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.262484 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.262920 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.262921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.262922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.262987 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.263171 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.263171 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.263172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.263312 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.263381 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.263557 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.263639 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 18:39:30.264510 17345 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.250.125:35565: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae"
member_type: NON_VOTER
last_known_addr {
host: "127.10.250.125"
port: 35565
}
attrs {
promote: true
}
}
dest_uuid: "91b96f5e5c59435abc16847a2b3db027"
cas_config_opid_index: -1
I20260620 18:39:30.264842 17345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: NON_VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: true } }
I20260620 18:39:30.265353 17905 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7d2401dd5ea84376b1824c16cdc174ae (127.10.250.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: NON_VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: true } } }
I20260620 18:39:30.265736 17851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:30.266345 17905 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.250.125:35565
I20260620 18:39:30.266616 17809 master.cc:537] Master@127.10.250.125:35565 shutting down...
I20260620 18:39:30.266710 17851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 FOLLOWER]: Refusing update from remote peer 91b96f5e5c59435abc16847a2b3db027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 18:39:30.266625 17904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "91b96f5e5c59435abc16847a2b3db027" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: NON_VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: true } } }
I20260620 18:39:30.266844 17904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:30.267045 17904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: NON_VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:39:30.267853 17332 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 18:39:30.268316 17332 consensus_peers.cc:487] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 -> Peer 7d2401dd5ea84376b1824c16cdc174ae (127.10.250.125:35565): Couldn't send request to peer 7d2401dd5ea84376b1824c16cdc174ae. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 18:39:30.269860 17809 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:30.269970 17809 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:30.270022 17809 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: stopping tablet replica
I20260620 18:39:30.282773 17809 master.cc:552] Master@127.10.250.125:35565 shutdown complete.
I20260620 18:39:30.282828 17809 master_runner.cc:305] Clearing existing system tablet
I20260620 18:39:30.282977 17809 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:30.283028 17809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:30.283694 17809 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:30.284324 17909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.284482 17809 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:30.284534 17809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:30.284556 17809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/wal
uuid: "7d2401dd5ea84376b1824c16cdc174ae"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-wp6g"
server_key: "df19bb2ad48a2b9df666213dbb4b7bd6"
server_key_iv: "2f270384820a58657d8f6a1a94f82f9f"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.285198 17809 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:30.287106 17809 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 18:39:30.287168 17809 log.cc:1196] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 18:39:30.287418 17809 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: Deleting consensus metadata
I20260620 18:39:30.287549 17809 master_runner.cc:315] Copying system tablet from 127.10.250.126:46691
I20260620 18:39:30.288201 17809 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: tablet copy: Beginning tablet copy session from remote peer at address 127.10.250.126:46691
I20260620 18:39:30.294669 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.288642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36770 (local address 127.10.250.126:46691)
0620 18:39:30.288697 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.288699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.288802 (+ 103us) server_negotiation.cc:407] Connection header received
0620 18:39:30.288891 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.288892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.288904 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.288964 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.289284 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.289802 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.290593 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.290808 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.290904 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.291907 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.291908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.291909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.291917 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.293397 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.293774 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.293775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.293775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.293840 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.294074 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.294075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.294075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.294199 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.294273 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.294456 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.294571 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 18:39:30.294687 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.288563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:36770)
0620 18:39:30.288694 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.288707 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.288820 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.288970 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.288971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.289148 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.289199 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.289201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.289904 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.289905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.290462 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.290464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.290905 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.290906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.290974 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.291475 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.291481 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.291802 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.293499 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.293500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.293501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.293688 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.293938 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.293939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.293940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.294008 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.294273 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.294274 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.294323 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.294426 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.294569 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":83,"threads_started":1}
I20260620 18:39:30.295260 17365 tablet_copy_service.cc:140] P 91b96f5e5c59435abc16847a2b3db027: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7d2401dd5ea84376b1824c16cdc174ae ({username='oryx', principal='oryx/127.10.250.125@KRBTEST.COM'} at 127.0.0.1:36770)
I20260620 18:39:30.295347 17365 tablet_copy_service.cc:161] P 91b96f5e5c59435abc16847a2b3db027: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7d2401dd5ea84376b1824c16cdc174ae at {username='oryx', principal='oryx/127.10.250.125@KRBTEST.COM'} at 127.0.0.1:36770: session id = 7d2401dd5ea84376b1824c16cdc174ae-00000000000000000000000000000000
I20260620 18:39:30.296128 17365 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:39:30.296876 17809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 18:39:30.298319 17809 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: tablet copy: Starting download of 0 data blocks...
I20260620 18:39:30.298482 17809 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: tablet copy: Starting download of 1 WAL segments...
I20260620 18:39:30.299376 17809 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:39:30.300299 17365 tablet_copy_service.cc:342] P 91b96f5e5c59435abc16847a2b3db027: Request end of tablet copy session 7d2401dd5ea84376b1824c16cdc174ae-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.250.125@KRBTEST.COM'} at 127.0.0.1:36770
I20260620 18:39:30.300361 17365 tablet_copy_service.cc:433] P 91b96f5e5c59435abc16847a2b3db027: ending tablet copy session 7d2401dd5ea84376b1824c16cdc174ae-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7d2401dd5ea84376b1824c16cdc174ae
I20260620 18:39:30.302855 17809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:30.304096 17917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.304180 17919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.304234 17916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:30.304364 17809 server_base.cc:1034] running on GCE node
I20260620 18:39:30.304471 17809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:30.304623 17809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:30.305765 17809 hybrid_clock.cc:648] HybridClock initialized: now 1781980770305756 us; error 25 us; skew 500 ppm
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:30.307276 17809 init.cc:377] Logged in from keytab as oryx/127.10.250.125@KRBTEST.COM (short username oryx)
I20260620 18:39:30.307531 17809 webserver.cc:460] Webserver started at http://127.10.250.125:41737/ using document root <none> and password file <none>
I20260620 18:39:30.307650 17809 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:30.307678 17809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:30.308386 17809 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:30.308935 17926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.309094 17809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:30.309147 17809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/wal
uuid: "7d2401dd5ea84376b1824c16cdc174ae"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-wp6g"
server_key: "df19bb2ad48a2b9df666213dbb4b7bd6"
server_key_iv: "2f270384820a58657d8f6a1a94f82f9f"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.309209 17809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:30.316581 17809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:30.318148 17809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:30.321142 17809 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.125:35565
I20260620 18:39:30.321584 17809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-1/data/info.pb
I20260620 18:39:30.321828 17978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.125:35565 every 8 connection(s)
I20260620 18:39:30.322372 17979 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:39:30.322849 17979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: Bootstrap starting.
I20260620 18:39:30.325066 17979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:39:30.325310 17979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: Bootstrap complete.
I20260620 18:39:30.325506 17979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: NON_VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: true } }
I20260620 18:39:30.325573 17979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7d2401dd5ea84376b1824c16cdc174ae, State: Initialized, Role: LEARNER
I20260620 18:39:30.325599 17979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: NON_VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: true } }
I20260620 18:39:30.325776 17979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:30.325776 17982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: NON_VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: true } } }
I20260620 18:39:30.325898 17982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: This master's current role is: LEARNER
I20260620 18:39:30.327291 17993 catalog_manager.cc:1259] Loaded cluster ID: a2dd3d882d904832ba1e2c8e352f2c66
I20260620 18:39:30.327370 17993 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: loading cluster ID for follower catalog manager: success
I20260620 18:39:30.328358 17993 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: acquiring CA information for follower catalog manager: success
I20260620 18:39:30.328636 17993 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:39:30.339319 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.331914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56976 (local address 127.10.250.125:35565)
0620 18:39:30.332041 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.332043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.332054 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:39:30.332082 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.332083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.332094 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.332191 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.332772 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.333759 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.334652 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.334965 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.335049 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.336300 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.336301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.336301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.336310 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.338025 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.338374 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.338375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.338376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.338446 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.338735 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.338736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.338736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.338902 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.339012 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.339111 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.339203 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for oryx/127.10.250.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.250.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.250.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.124.keytab.
Entry for principal oryx/127.10.250.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.124.keytab.
I20260620 18:39:30.365708 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.124:35477
--webserver_interface=127.10.250.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.124.keytab
--principal=oryx/127.10.250.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:46691,127.10.250.125:35565,127.10.250.124:35477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:30.449939 17997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:30.450155 17997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:30.450210 17997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:30.451462 17997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:30.451505 17997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:30.451520 17997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:30.451534 17997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:30.451545 17997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:30.452960 17997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:46691,127.10.250.125:35565,127.10.250.124:35477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.124.keytab
--principal=oryx/127.10.250.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.124:35477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.250.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:30.453158 17997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:30.453330 17997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:30.458218 18005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.458238 18003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:30.458477 17997 server_base.cc:1034] running on GCE node
W20260620 18:39:30.458247 18002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:30.458781 17997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:30.459030 17997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:30.460171 17997 hybrid_clock.cc:648] HybridClock initialized: now 1781980770460151 us; error 44 us; skew 500 ppm
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:30.462834 17997 init.cc:377] Logged in from keytab as oryx/127.10.250.124@KRBTEST.COM (short username oryx)
I20260620 18:39:30.463339 17997 webserver.cc:460] Webserver started at http://127.10.250.124:46761/ using document root <none> and password file <none>
I20260620 18:39:30.463594 17997 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:30.463641 17997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:30.463833 17997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:30.464929 17997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/data/instance:
uuid: "834ae29701f044b2a8e99747efffa486"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-wp6g"
server_key: "65fcb8b4e298557a24bcfbba7856bee7"
server_key_iv: "6f76671e671f430718fbdfbe4bfb6e9f"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.465305 17997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/wal/instance:
uuid: "834ae29701f044b2a8e99747efffa486"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-wp6g"
server_key: "65fcb8b4e298557a24bcfbba7856bee7"
server_key_iv: "6f76671e671f430718fbdfbe4bfb6e9f"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.466760 17997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:39:30.467622 18012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.467829 17997 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:30.467924 17997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/wal
uuid: "834ae29701f044b2a8e99747efffa486"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-wp6g"
server_key: "65fcb8b4e298557a24bcfbba7856bee7"
server_key_iv: "6f76671e671f430718fbdfbe4bfb6e9f"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.467995 17997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:30.478014 17997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:30.480414 17997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:30.480579 17997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:30.485050 17997 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.124:35477
I20260620 18:39:30.485085 18064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.124:35477 every 8 connection(s)
I20260620 18:39:30.485518 17997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/data/info.pb
I20260620 18:39:30.486581 18065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:30.488163 18065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:39:30.490911 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 17997
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.124@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.250.124@KRBTEST.COM
I20260620 18:39:30.499697 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.489083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.10.250.126:46691)
0620 18:39:30.489179 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.489181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.489395 (+ 214us) server_negotiation.cc:407] Connection header received
0620 18:39:30.489930 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.489932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.489946 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.489997 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.490693 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.491277 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.492366 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.492559 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.492630 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.495606 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.495607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.495608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.495617 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.498036 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.498580 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.498582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.498582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.498644 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.498896 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.498896 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.498897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.499074 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.499142 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.499505 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.499597 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:39:30.499913 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.488966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34042)
0620 18:39:30.489215 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.489268 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.489812 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.490066 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.490070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.490334 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.490595 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.490600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.491367 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.491368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.492251 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.492254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.492669 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.492670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.492786 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.493902 (+ 1116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.493911 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.495517 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.498120 (+ 2603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.498127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.498129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.498467 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.498712 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.498712 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.498712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.498816 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.499149 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.499154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.499328 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.499445 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.499628 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":116,"threads_started":1}
I20260620 18:39:30.500507 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.491431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.10.250.124:35477)
0620 18:39:30.491606 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.491610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.491632 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:39:30.491653 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.491654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.491680 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.491763 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.492333 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.493114 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.493864 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.494166 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.494355 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.497244 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.497253 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.497253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.497270 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.499054 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.499476 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.499479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.499480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.499542 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.499760 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.499761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.499761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.499951 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.500147 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.500290 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.500370 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260620 18:39:30.500674 18065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.124@KRBTEST.COM for oryx/127.10.250.125@KRBTEST.COM
I20260620 18:39:30.507701 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.501023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:35565 (local address 127.0.0.1:56980)
0620 18:39:30.501057 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.501069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.501103 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.501263 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.501264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.501374 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.501409 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.501411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.502032 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.502033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.502717 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.502719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.503128 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.503129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.503179 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.503689 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.503694 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.505093 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.506793 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.506794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.506794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.506975 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.507193 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.507194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.507194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.507244 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.507464 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.507464 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.507502 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.507554 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.507631 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 18:39:30.507858 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.501098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56980 (local address 127.10.250.125:35565)
0620 18:39:30.501138 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.501140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.501147 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:39:30.501175 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.501176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.501190 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.501255 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.501458 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.501935 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.502827 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.503046 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.503118 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.505190 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.505192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.505192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.505201 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.506693 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.507063 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.507065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.507065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.507124 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.507301 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.507302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.507302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.507410 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.507462 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.507635 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.507709 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:39:30.508273 18065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.124@KRBTEST.COM for oryx/127.10.250.124@KRBTEST.COM
I20260620 18:39:30.514914 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.508584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:35477 (local address 127.0.0.1:55436)
0620 18:39:30.508640 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.508651 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.508690 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.508798 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.508799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.508913 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.508945 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.508947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.509615 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.509616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.510083 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.510084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.510610 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.510611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.510658 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.511101 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.511106 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.512364 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.513772 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.513773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.513773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.513941 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.514164 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.514165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.514165 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.514225 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.514616 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.514617 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.514662 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.514729 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.514815 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 18:39:30.514963 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.508642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.10.250.124:35477)
0620 18:39:30.508678 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.508679 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.508686 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:39:30.508735 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.508736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.508745 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.508794 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.508985 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.509544 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.510168 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.510475 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.510607 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.512469 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.512470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.512470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.512477 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.513696 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.514019 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.514020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.514020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.514073 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.514328 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.514329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.514330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.514442 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.514508 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.514757 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.514864 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":19}
I20260620 18:39:30.516463 18065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: Bootstrap starting.
I20260620 18:39:30.517315 18065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:30.517604 18065 log.cc:824] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:30.518349 18065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: No bootstrap required, opened a new log
I20260620 18:39:30.519860 18065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } }
I20260620 18:39:30.519986 18065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:30.520009 18065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834ae29701f044b2a8e99747efffa486, State: Initialized, Role: FOLLOWER
I20260620 18:39:30.520156 18065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } }
I20260620 18:39:30.520567 18065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:30.520529 18070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } } }
I20260620 18:39:30.520665 18070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 18:39:30.522353 18081 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:39:30.522419 18081 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:39:30.524825 17997 master_runner.cc:428] Detected that this master 834ae29701f044b2a8e99747efffa486 is joining an existing cluster
I20260620 18:39:30.524899 17997 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.250.124:35477
I20260620 18:39:30.533599 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.526381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.10.250.125:35565)
0620 18:39:30.526421 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.526422 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.526608 (+ 186us) server_negotiation.cc:407] Connection header received
0620 18:39:30.526709 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.526710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.526721 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.526754 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.527257 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.528093 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.529423 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.529733 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.529777 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.530926 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.530927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.530928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.530935 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.532283 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.532737 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.532738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.532739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.532815 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.533070 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.533071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.533071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.533190 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.533248 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.533458 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.533532 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:39:30.533671 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.526355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:35565 (local address 127.0.0.1:56988)
0620 18:39:30.526507 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.526557 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.526636 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.526824 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.526824 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.527096 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.527163 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.527165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.528240 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.528241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.529342 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.529344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.529866 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.529867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.529939 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.530297 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.530303 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.530805 (+ 502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.532369 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.532370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.532371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.532609 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.532903 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.532904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.532905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.532974 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.533263 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.533264 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.533341 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.533432 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.533549 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260620 18:39:30.546252 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.525902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34054 (local address 127.10.250.126:46691)
0620 18:39:30.525945 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.525947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.526109 (+ 162us) server_negotiation.cc:407] Connection header received
0620 18:39:30.526158 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.526158 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.526170 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.526219 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.526733 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.527560 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.528652 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.528964 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.529001 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.543052 (+ 14051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.543054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.543054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.543064 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.544800 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.545361 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.545363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.545363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.545430 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.545636 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.545637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.545637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.545786 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.545857 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.546055 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.546144 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:39:30.546265 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.525863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34054)
0620 18:39:30.526006 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.526016 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.526098 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.526220 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.526221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.526540 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.526640 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.526643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.527649 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.527650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.528561 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.528563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.529079 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.529079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.529148 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.529752 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.529758 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.542929 (+ 13171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.544897 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.544899 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.544899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.545270 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.545504 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.545504 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.545505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.545567 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.545871 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.545879 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.545925 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.546017 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.546134 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":4,"thread_start_us":91,"threads_started":1}
I20260620 18:39:30.546715 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.526144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.10.250.124:35477)
0620 18:39:30.526183 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.526185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.526564 (+ 379us) server_negotiation.cc:407] Connection header received
0620 18:39:30.526651 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.526652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.526660 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.526686 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.527160 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.527851 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.528859 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.529185 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.529273 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.543883 (+ 14610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.543885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.543885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.543893 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.545379 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.545742 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.545743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.545743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.545786 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.545989 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.545990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.545990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.546106 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.546163 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.546572 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.546646 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":19}
I20260620 18:39:30.547108 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.526153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:35477 (local address 127.0.0.1:55446)
0620 18:39:30.526483 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.526493 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.526580 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.526737 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.526738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.526968 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.527070 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.527073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.527986 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.527987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.528787 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.528789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.529284 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.529284 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.529413 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.543328 (+ 13915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.543333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.543801 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.545458 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.545459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.545459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.545669 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.545853 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.545854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.545854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.545916 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.546186 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.546186 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.546233 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.546912 (+ 679us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.547007 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":69,"threads_started":1}
I20260620 18:39:30.551833 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 17407
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980769, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.126@KRBTEST.COM for oryx/127.10.250.124@KRBTEST.COM
I20260620 18:39:30.556162 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.548652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:35477 (local address 127.0.0.1:55448)
0620 18:39:30.548705 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.548717 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.548758 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.548891 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.548891 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.549016 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.549058 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.549059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.549951 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.549952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.550482 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.550484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.550956 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.550957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.551031 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.551490 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.551496 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.553047 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.554997 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.554998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.554999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.555229 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.555418 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.555419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.555419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.555477 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.555757 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.555758 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.555805 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.555865 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.556000 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 18:39:30.556520 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.548730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55448 (local address 127.10.250.124:35477)
0620 18:39:30.548769 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.548771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.548779 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:39:30.548813 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.548814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.548825 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.548889 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.549122 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.549854 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.550619 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.550866 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.550955 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.553141 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.553143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.553143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.553151 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.554903 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.555301 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.555302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.555303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.555355 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.555534 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.555535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.555535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.555678 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.555733 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.556299 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.556381 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:39:30.559237 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:33437
--local_ip_for_outbound_sockets=127.10.250.65
--tserver_master_addrs=127.10.250.126:46691
--webserver_port=43523
--webserver_interface=127.10.250.65
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.65.keytab
--principal=oryx/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.250.126:46691,127.10.250.125:35565
--tserver_master_addrs=127.10.250.126:46691,127.10.250.125:35565,127.10.250.124:35477 with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:39:30.559459 17345 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.250.124:35477: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "834ae29701f044b2a8e99747efffa486"
member_type: NON_VOTER
last_known_addr {
host: "127.10.250.124"
port: 35477
}
attrs {
promote: true
}
}
dest_uuid: "91b96f5e5c59435abc16847a2b3db027"
cas_config_opid_index: 5
I20260620 18:39:30.559725 17345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: NON_VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: true } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } }
I20260620 18:39:30.560261 17904 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 834ae29701f044b2a8e99747efffa486 (127.10.250.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: NON_VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: true } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } } }
I20260620 18:39:30.560985 18039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:30.561421 17904 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.250.124:35477
I20260620 18:39:30.561493 17904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "91b96f5e5c59435abc16847a2b3db027" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: NON_VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: true } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } } }
I20260620 18:39:30.561558 17904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:30.561847 17997 master.cc:537] Master@127.10.250.124:35477 shutting down...
I20260620 18:39:30.562057 18039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 1 FOLLOWER]: Refusing update from remote peer 91b96f5e5c59435abc16847a2b3db027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 18:39:30.562471 17905 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:39:30.563400 17329 consensus_peers.cc:487] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 -> Peer 834ae29701f044b2a8e99747efffa486 (127.10.250.124:35477): Couldn't send request to peer 834ae29701f044b2a8e99747efffa486. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 18:39:30.564986 17997 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:30.565088 17997 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:30.565141 17997 tablet_replica.cc:330] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: stopping tablet replica
I20260620 18:39:30.567154 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.560581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.10.250.125:35565)
0620 18:39:30.560622 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.560624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.560961 (+ 337us) server_negotiation.cc:407] Connection header received
0620 18:39:30.561108 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.561109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.561134 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.561177 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:30.561483 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.562298 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.565996 (+ 3698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.566657 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.566711 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.566783 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.567026 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.567083 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:39:30.567701 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.560482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:35565 (local address 127.0.0.1:57004)
0620 18:39:30.560901 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.560914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.560990 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.561316 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.561316 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.561318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:30.561404 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.561406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.562852 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.562853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.565872 (+ 3019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.565877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.566849 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.566850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.566917 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.566946 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.567568 (+ 622us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.567625 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399}
I20260620 18:39:30.568011 17953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 LEARNER]: Refusing update from remote peer 91b96f5e5c59435abc16847a2b3db027: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 18:39:30.568233 18090 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: NON_VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 18:39:30.568868 17953 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 834ae29701f044b2a8e99747efffa486 (127.10.250.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: NON_VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: true } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } } }
I20260620 18:39:30.570255 17982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91b96f5e5c59435abc16847a2b3db027. Latest consensus state: current_term: 1 leader_uuid: "91b96f5e5c59435abc16847a2b3db027" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: NON_VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: true } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } } }
I20260620 18:39:30.570363 17982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: This master's current role is: LEARNER
I20260620 18:39:30.570406 17982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91b96f5e5c59435abc16847a2b3db027" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: NON_VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: true } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } } }
I20260620 18:39:30.570649 18090 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027: attempting to promote NON_VOTER 7d2401dd5ea84376b1824c16cdc174ae to VOTER
I20260620 18:39:30.570914 18090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } }
I20260620 18:39:30.571389 17953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 LEARNER]: Refusing update from remote peer 91b96f5e5c59435abc16847a2b3db027: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 18:39:30.571676 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.571539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:35477 (local address 127.0.0.1:55450)
0620 18:39:30.571590 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.571638 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:35477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 18:39:30.571760 17905 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 18:39:30.570467 17982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: This master's current role is: LEARNER
I20260620 18:39:30.572373 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.572279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:35477 (local address 127.0.0.1:55466)
0620 18:39:30.572321 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.572345 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:35477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 18:39:30.572500 17329 consensus_peers.cc:487] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 -> Peer 834ae29701f044b2a8e99747efffa486 (127.10.250.124:35477): Couldn't send request to peer 834ae29701f044b2a8e99747efffa486. Status: Network error: Client connection negotiation failed: client connection to 127.10.250.124:35477: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 18:39:30.572479 17904 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7d2401dd5ea84376b1824c16cdc174ae (127.10.250.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } } }
I20260620 18:39:30.573585 17904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "91b96f5e5c59435abc16847a2b3db027" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } } }
I20260620 18:39:30.573686 17904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:30.573747 17904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91b96f5e5c59435abc16847a2b3db027" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } } }
I20260620 18:39:30.573819 17904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:30.574226 17953 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7d2401dd5ea84376b1824c16cdc174ae (127.10.250.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } } }
I20260620 18:39:30.575340 17982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91b96f5e5c59435abc16847a2b3db027" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } } }
I20260620 18:39:30.575431 17982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:39:30.578755 17997 master.cc:552] Master@127.10.250.124:35477 shutdown complete.
I20260620 18:39:30.578820 17997 master_runner.cc:305] Clearing existing system tablet
I20260620 18:39:30.578992 17997 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:30.579043 17997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:30.579825 17997 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:30.580406 18095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.580574 17997 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:30.580639 17997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.580665 17997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/wal
uuid: "834ae29701f044b2a8e99747efffa486"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-wp6g"
server_key: "65fcb8b4e298557a24bcfbba7856bee7"
server_key_iv: "6f76671e671f430718fbdfbe4bfb6e9f"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.581465 17997 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:30.583045 17997 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 18:39:30.583124 17997 log.cc:1196] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 18:39:30.583333 17997 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: Deleting consensus metadata
I20260620 18:39:30.583456 17997 master_runner.cc:315] Copying system tablet from 127.10.250.126:46691
I20260620 18:39:30.584124 17997 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: tablet copy: Beginning tablet copy session from remote peer at address 127.10.250.126:46691
I20260620 18:39:30.591449 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.584618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34066 (local address 127.10.250.126:46691)
0620 18:39:30.584665 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.584667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.584678 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:39:30.584736 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.584736 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.584748 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.584808 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.585224 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.586000 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.587093 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.587379 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.587419 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.588623 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.588624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.588624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.588632 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.590168 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.590649 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.590650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.590651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.590709 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.590914 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.590915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.590915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.591043 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.591084 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.591275 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.591357 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:39:30.591907 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.584437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34066)
0620 18:39:30.584580 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.584594 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.584669 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.584815 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.584816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.585039 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.585129 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.585131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.586123 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.586124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.586988 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.586991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.587502 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.587503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.587564 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.588109 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.588114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.588519 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.590256 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.590257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.590257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.590460 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.590778 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.590779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.590779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.590835 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.591126 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.591127 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.591171 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.591688 (+ 517us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.591797 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260620 18:39:30.592222 17365 tablet_copy_service.cc:140] P 91b96f5e5c59435abc16847a2b3db027: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 834ae29701f044b2a8e99747efffa486 ({username='oryx', principal='oryx/127.10.250.124@KRBTEST.COM'} at 127.0.0.1:34066)
I20260620 18:39:30.592275 17365 tablet_copy_service.cc:161] P 91b96f5e5c59435abc16847a2b3db027: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 834ae29701f044b2a8e99747efffa486 at {username='oryx', principal='oryx/127.10.250.124@KRBTEST.COM'} at 127.0.0.1:34066: session id = 834ae29701f044b2a8e99747efffa486-00000000000000000000000000000000
I20260620 18:39:30.592854 17365 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:39:30.593384 17997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 18:39:30.594784 17997 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: tablet copy: Starting download of 0 data blocks...
I20260620 18:39:30.594925 17997 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: tablet copy: Starting download of 1 WAL segments...
I20260620 18:39:30.595595 17997 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:39:30.596465 17365 tablet_copy_service.cc:342] P 91b96f5e5c59435abc16847a2b3db027: Request end of tablet copy session 834ae29701f044b2a8e99747efffa486-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.250.124@KRBTEST.COM'} at 127.0.0.1:34066
I20260620 18:39:30.596518 17365 tablet_copy_service.cc:433] P 91b96f5e5c59435abc16847a2b3db027: ending tablet copy session 834ae29701f044b2a8e99747efffa486-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 834ae29701f044b2a8e99747efffa486
I20260620 18:39:30.599731 17997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:30.601059 18103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.601184 18105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.601176 18102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:30.603590 17997 server_base.cc:1034] running on GCE node
I20260620 18:39:30.603700 17997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:30.603847 17997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:30.604991 17997 hybrid_clock.cc:648] HybridClock initialized: now 1781980770604982 us; error 23 us; skew 500 ppm
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:30.606482 17997 init.cc:377] Logged in from keytab as oryx/127.10.250.124@KRBTEST.COM (short username oryx)
I20260620 18:39:30.606776 17997 webserver.cc:460] Webserver started at http://127.10.250.124:45039/ using document root <none> and password file <none>
I20260620 18:39:30.606907 17997 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:30.606940 17997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:30.607731 17997 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:30.608326 18112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.608441 17997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:30.608479 17997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/wal
uuid: "834ae29701f044b2a8e99747efffa486"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-wp6g"
server_key: "65fcb8b4e298557a24bcfbba7856bee7"
server_key_iv: "6f76671e671f430718fbdfbe4bfb6e9f"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.608541 17997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:30.627271 17997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:30.629220 17997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:30.632323 17997 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.124:35477
I20260620 18:39:30.632663 17997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/master-2/data/info.pb
I20260620 18:39:30.634630 18164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.124:35477 every 8 connection(s)
I20260620 18:39:30.635463 18165 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:39:30.635977 18165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: Bootstrap starting.
I20260620 18:39:30.638885 18165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:39:30.639108 18165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: Bootstrap complete.
I20260620 18:39:30.639298 18165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } }
I20260620 18:39:30.639386 18165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 834ae29701f044b2a8e99747efffa486, State: Initialized, Role: LEARNER
I20260620 18:39:30.639420 18165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } }
I20260620 18:39:30.639662 18165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:30.639655 18168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } } }
I20260620 18:39:30.639909 18168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:39:30.641218 18179 catalog_manager.cc:1259] Loaded cluster ID: a2dd3d882d904832ba1e2c8e352f2c66
I20260620 18:39:30.641425 18179 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: loading cluster ID for follower catalog manager: success
I20260620 18:39:30.642906 18179 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: acquiring CA information for follower catalog manager: success
I20260620 18:39:30.643224 18179 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 18:39:30.662140 18091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:30.662369 18091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:30.662418 18091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:30.663717 18091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:30.663791 18091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:30.663839 18091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:30.665419 18091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.65.keytab
--principal=oryx/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:33437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=43523
--tserver_master_addrs=127.10.250.126:46691,127.10.250.125:35565,127.10.250.124:35477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:30.665701 18091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:30.665951 18091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:30.671955 18184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.671972 18187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.671968 18185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:30.672683 18091 server_base.cc:1034] running on GCE node
I20260620 18:39:30.672891 18091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:30.673164 18091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:30.674332 18091 hybrid_clock.cc:648] HybridClock initialized: now 1781980770674316 us; error 27 us; skew 500 ppm
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:30.677652 18091 init.cc:377] Logged in from keytab as oryx/127.10.250.65@KRBTEST.COM (short username oryx)
I20260620 18:39:30.678124 18091 webserver.cc:460] Webserver started at http://127.10.250.65:43523/ using document root <none> and password file <none>
I20260620 18:39:30.678392 18091 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:30.678489 18091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:30.680006 18091 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:30.680739 18194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.680953 18091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:30.681073 18091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c"
format_stamp: "Formatted at 2026-06-20 18:39:29 on dist-test-slave-wp6g"
server_key: "454929df5abe12e7a67fb95e0b2de5be"
server_key_iv: "0623a177857738a9d040ec475b72ad6e"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.681500 18091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:30.692003 18091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:30.694203 18091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:30.694397 18091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:30.694717 18091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:30.695051 18091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:30.695106 18091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.695150 18091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:30.695176 18091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.701664 18091 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:33437
I20260620 18:39:30.701740 18307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:33437 every 8 connection(s)
I20260620 18:39:30.702104 18091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:30.707875 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 18091
I20260620 18:39:30.708048 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 17541
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.125@KRBTEST.COM
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.124@KRBTEST.COM
I20260620 18:39:30.713330 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:37457
--local_ip_for_outbound_sockets=127.10.250.66
--tserver_master_addrs=127.10.250.126:46691
--webserver_port=35867
--webserver_interface=127.10.250.66
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.66.keytab
--principal=oryx/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.250.126:46691,127.10.250.125:35565
--tserver_master_addrs=127.10.250.126:46691,127.10.250.125:35565,127.10.250.124:35477 with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM
I20260620 18:39:30.717391 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.703094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:36749 (local address 127.10.250.125:35565)
0620 18:39:30.703178 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.703181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.703931 (+ 750us) server_negotiation.cc:407] Connection header received
0620 18:39:30.703991 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.703992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.704005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.704065 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.704898 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.705406 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.706856 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.707152 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.707219 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.714000 (+ 6781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.714002 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.714003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.714010 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.715769 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.716259 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.716260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.716260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.716318 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.716609 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.716626 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.716626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.716793 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.716864 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.717187 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.717270 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:39:30.717612 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.702905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:35477 (local address 127.10.250.65:44211)
0620 18:39:30.703109 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.703156 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.703730 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.704179 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.704180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.704563 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.704796 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.704803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.705973 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.705974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.706664 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.706666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.707139 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.707139 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.707345 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.708299 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.708303 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.713356 (+ 5053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.715499 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.715507 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.715510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.716166 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.716391 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.716392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.716392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.716549 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.716840 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.716846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.717022 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.717141 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.717328 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":4,"thread_start_us":117,"threads_started":1}
I20260620 18:39:30.717732 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.702905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.65:48103)
0620 18:39:30.703322 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.703331 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.703744 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.703985 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.703988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.704571 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.704796 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.704803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.705761 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.705762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.706721 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.706724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.707534 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.707535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.707603 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.708173 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.708178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.714095 (+ 5917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.716609 (+ 2514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.716610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.716610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.716822 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.717027 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.717027 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.717028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.717093 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.717385 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.717386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.717424 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.717489 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.717624 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":35,"thread_start_us":95,"threads_started":1}
I20260620 18:39:30.717970 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.702905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:35565 (local address 127.10.250.65:36749)
0620 18:39:30.703862 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.703870 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.703937 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.704124 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.704125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.704554 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.704795 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.704801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.705512 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.705513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.706727 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.706728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.707270 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.707271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.707345 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.708013 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.708024 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.713874 (+ 5850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.715862 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.715863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.715863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.716166 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.716392 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.716392 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.716393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.716549 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.716894 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.716895 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.717023 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.717516 (+ 493us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.717856 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":416,"thread_start_us":68,"threads_started":1}
I20260620 18:39:30.718003 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.702982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:44211 (local address 127.10.250.124:35477)
0620 18:39:30.704043 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.704045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.704053 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:39:30.704076 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.704077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.704089 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.704152 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.704913 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.705891 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.706755 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.707055 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.707130 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.713516 (+ 6386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.713519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.713519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.713531 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.715405 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.716262 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.716264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.716264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.716318 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.716627 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.716628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.716628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.716759 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.716836 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.717762 (+ 926us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.717862 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1038,"thread_start_us":63,"threads_started":1}
I20260620 18:39:30.718385 18308 heartbeater.cc:347] Connected to a master server at 127.10.250.124:35477
I20260620 18:39:30.718472 18308 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:30.718674 18308 heartbeater.cc:511] Master 127.10.250.124:35477 requested a full tablet report, sending...
I20260620 18:39:30.718849 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.703161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:48103 (local address 127.10.250.126:46691)
0620 18:39:30.703206 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.703208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.703403 (+ 195us) server_negotiation.cc:407] Connection header received
0620 18:39:30.703835 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.703836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.703848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.703900 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.704914 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.705679 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.707271 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.707456 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.707534 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.714180 (+ 6646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.714181 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.714181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.714191 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.716515 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.716902 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.716903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.716904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.716960 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.717160 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.717161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.717161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.717299 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.717358 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.718618 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.718711 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:39:30.718933 18310 heartbeater.cc:347] Connected to a master server at 127.10.250.125:35565
I20260620 18:39:30.718976 18310 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:30.719038 18310 heartbeater.cc:511] Master 127.10.250.125:35565 requested a full tablet report, sending...
I20260620 18:39:30.719136 18309 heartbeater.cc:347] Connected to a master server at 127.10.250.126:46691
I20260620 18:39:30.719190 18309 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:30.719197 18129 ts_manager.cc:194] Registered new tserver with Master: c4ef94b55fa94b9c90a32215dfb2fa2c (127.10.250.65:33437)
I20260620 18:39:30.719254 18309 heartbeater.cc:511] Master 127.10.250.126:46691 requested a full tablet report, sending...
I20260620 18:39:30.719377 17943 ts_manager.cc:194] Registered new tserver with Master: c4ef94b55fa94b9c90a32215dfb2fa2c (127.10.250.65:33437)
I20260620 18:39:30.719521 17345 ts_manager.cc:194] Re-registered known tserver with Master: c4ef94b55fa94b9c90a32215dfb2fa2c (127.10.250.65:33437)
I20260620 18:39:30.720091 17345 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:48103
W20260620 18:39:30.802759 18316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:30.802973 18316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:30.802999 18316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:30.804150 18316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:30.804188 18316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:30.804216 18316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:30.805644 18316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.66.keytab
--principal=oryx/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:37457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=35867
--tserver_master_addrs=127.10.250.126:46691,127.10.250.125:35565,127.10.250.124:35477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:30.805838 18316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:30.806011 18316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:30.811172 18324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.811373 18321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.811498 18322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:30.811738 18316 server_base.cc:1034] running on GCE node
I20260620 18:39:30.811991 18316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:30.812269 18316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:30.813428 18316 hybrid_clock.cc:648] HybridClock initialized: now 1781980770813384 us; error 56 us; skew 500 ppm
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:30.816063 18316 init.cc:377] Logged in from keytab as oryx/127.10.250.66@KRBTEST.COM (short username oryx)
I20260620 18:39:30.816520 18316 webserver.cc:460] Webserver started at http://127.10.250.66:35867/ using document root <none> and password file <none>
I20260620 18:39:30.816768 18316 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:30.816865 18316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:30.818295 18316 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:39:30.819002 18331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.819175 18316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:39:30.819284 18316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "e4e832d399164153a9c69cae14100b88"
format_stamp: "Formatted at 2026-06-20 18:39:29 on dist-test-slave-wp6g"
server_key: "59826aae2fc257e1a30ba2111e0f13a4"
server_key_iv: "f46da311b6a0ebe3caa221817f069894"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.819648 18316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:30.836196 18316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:30.838207 18316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:30.838376 18316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:30.838668 18316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:30.839000 18316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:30.839054 18316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.839098 18316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:30.839124 18316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.845312 18316 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:37457
I20260620 18:39:30.845351 18444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:37457 every 8 connection(s)
I20260620 18:39:30.845762 18316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:30.851647 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 18316
I20260620 18:39:30.851835 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 17675
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.125@KRBTEST.COM
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.124@KRBTEST.COM
I20260620 18:39:30.858026 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:36941
--local_ip_for_outbound_sockets=127.10.250.67
--tserver_master_addrs=127.10.250.126:46691
--webserver_port=41451
--webserver_interface=127.10.250.67
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.67.keytab
--principal=oryx/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.250.126:46691,127.10.250.125:35565
--tserver_master_addrs=127.10.250.126:46691,127.10.250.125:35565,127.10.250.124:35477 with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:39:30.858914 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.846599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:56729 (local address 127.10.250.126:46691)
0620 18:39:30.846814 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.846816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.847136 (+ 320us) server_negotiation.cc:407] Connection header received
0620 18:39:30.848084 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.848085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.848098 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.848132 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.848859 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.849369 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.850911 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.851239 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.851328 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.855227 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.855229 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.855229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.855238 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.857106 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.857666 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.857668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.857668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.857729 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.858030 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.858031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.858032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.858184 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.858239 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.858674 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.858769 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260620 18:39:30.859395 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.846934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:53161 (local address 127.10.250.125:35565)
0620 18:39:30.846976 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.846978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.846989 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:39:30.848059 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.848060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.848074 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.848132 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.848821 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.849603 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.851464 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.851770 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.851820 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.855227 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.855228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.855229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.855238 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.857582 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.858494 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.858496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.858496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.858588 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.858806 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.858807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.858807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.858939 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.858996 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.859195 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.859293 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:39:30.859675 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.846568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.66:56729)
0620 18:39:30.847051 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.847062 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.847974 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.848218 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.848221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.848667 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.848759 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.848762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.849470 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.849471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.850819 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.850821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.851375 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.851376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.851599 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.852230 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.852241 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.855100 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.857211 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.857220 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.857222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.857580 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.857803 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.857804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.857804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.857951 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.858272 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.858279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.858485 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.859052 (+ 567us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.859269 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":21,"thread_start_us":94,"threads_started":1}
I20260620 18:39:30.859799 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.846568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:35565 (local address 127.10.250.66:53161)
0620 18:39:30.846773 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.846827 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.847947 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.848211 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.848219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.848523 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.848720 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.848724 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.850612 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.850613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.851366 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.851369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.851896 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.851897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.852903 (+ 1006us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.853410 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.853416 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.855104 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.857954 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.857955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.857955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.858406 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.858667 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.858668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.858668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.858732 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.859031 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.859032 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.859078 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.859473 (+ 395us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.859596 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":101,"threads_started":1}
I20260620 18:39:30.860251 18446 heartbeater.cc:347] Connected to a master server at 127.10.250.126:46691
I20260620 18:39:30.860251 18447 heartbeater.cc:347] Connected to a master server at 127.10.250.125:35565
I20260620 18:39:30.860406 18446 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:30.860405 18447 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:30.860643 18446 heartbeater.cc:511] Master 127.10.250.126:46691 requested a full tablet report, sending...
I20260620 18:39:30.860646 18447 heartbeater.cc:511] Master 127.10.250.125:35565 requested a full tablet report, sending...
I20260620 18:39:30.861135 17345 ts_manager.cc:194] Re-registered known tserver with Master: e4e832d399164153a9c69cae14100b88 (127.10.250.66:37457)
I20260620 18:39:30.861167 17943 ts_manager.cc:194] Registered new tserver with Master: e4e832d399164153a9c69cae14100b88 (127.10.250.66:37457)
I20260620 18:39:30.861709 17345 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:56729
I20260620 18:39:30.863296 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.846568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:35477 (local address 127.10.250.66:57339)
0620 18:39:30.847159 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.847168 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.847974 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.848218 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.848221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.848667 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.848759 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.848762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.849691 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.849692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.850819 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.850821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.855095 (+ 4274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.855097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.855183 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.855616 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.855621 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.857237 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.859645 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.859646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.859646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.860424 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.862592 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.862593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.862593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.862656 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.863031 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.863032 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.863070 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.863128 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.863226 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":228,"thread_start_us":102,"threads_started":1}
I20260620 18:39:30.863451 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.846678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:57339 (local address 127.10.250.124:35477)
0620 18:39:30.846729 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.846732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.847236 (+ 504us) server_negotiation.cc:407] Connection header received
0620 18:39:30.848085 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.848086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.848103 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.848133 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.848870 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.849602 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.851400 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.851705 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.851766 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.857364 (+ 5598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.857366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.857367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.857376 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.859193 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.860568 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.860570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.860570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.860634 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.862759 (+ 2125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.862761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.862761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.862934 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.863018 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.863198 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.863316 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 18:39:30.863940 18445 heartbeater.cc:347] Connected to a master server at 127.10.250.124:35477
I20260620 18:39:30.864004 18445 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:30.864087 18445 heartbeater.cc:511] Master 127.10.250.124:35477 requested a full tablet report, sending...
I20260620 18:39:30.864414 18129 ts_manager.cc:194] Registered new tserver with Master: e4e832d399164153a9c69cae14100b88 (127.10.250.66:37457)
W20260620 18:39:30.943686 18452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:30.943897 18452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:30.943921 18452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:30.945149 18452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:30.945197 18452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:30.945225 18452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:30.946764 18452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.67.keytab
--principal=oryx/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:36941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=41451
--tserver_master_addrs=127.10.250.126:46691,127.10.250.125:35565,127.10.250.124:35477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:30.947024 18452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:30.947278 18452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:30.952409 18460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.952461 18457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.952472 18458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:30.952695 18452 server_base.cc:1034] running on GCE node
I20260620 18:39:30.952888 18452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:30.953099 18452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:30.954236 18452 hybrid_clock.cc:648] HybridClock initialized: now 1781980770954217 us; error 34 us; skew 500 ppm
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:30.956775 18452 init.cc:377] Logged in from keytab as oryx/127.10.250.67@KRBTEST.COM (short username oryx)
I20260620 18:39:30.957226 18452 webserver.cc:460] Webserver started at http://127.10.250.67:41451/ using document root <none> and password file <none>
I20260620 18:39:30.957490 18452 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:30.957576 18452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:30.959074 18452 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:39:30.959796 18467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.960058 18452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:30.960165 18452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-wp6g"
server_key: "2d32dc1042b7fee2a68a813a8176af54"
server_key_iv: "3bb81a9626182764f842cf624326c149"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.960530 18452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:30.971623 18452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:30.974109 18452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:30.974278 18452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:30.974568 18452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:30.974911 18452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:30.974964 18452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.975097 18452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:30.975159 18452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.981325 18452 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:36941
I20260620 18:39:30.981369 18580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:36941 every 8 connection(s)
I20260620 18:39:30.981729 18452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:30.984354 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 18452
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.125@KRBTEST.COM
I20260620 18:39:30.995970 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.982829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:56473 (local address 127.10.250.125:35565)
0620 18:39:30.982870 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.982872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.982883 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:39:30.983630 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.983631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.983644 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.983679 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.984800 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.985585 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.987142 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.987452 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.987487 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.991994 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.991996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.991996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.992006 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.994054 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.994751 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.994752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.994752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.994813 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.995127 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.995128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.995129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.995285 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.995348 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.995790 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.995869 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 18:39:30 dist-test-slave-wp6g krb5kdc[17300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.124@KRBTEST.COM
I20260620 18:39:30.997069 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.982468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:35565 (local address 127.10.250.67:56473)
0620 18:39:30.982708 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.982760 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.983509 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.983773 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.983774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.984343 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.984681 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.984688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.985703 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.985704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.987028 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.987029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.987583 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.987583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.987736 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.988529 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.988540 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.991882 (+ 3342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.994147 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.994158 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.994161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.994560 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.994880 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.994881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.994881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.995031 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.995380 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.995389 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.995621 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.995755 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.995941 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":90,"thread_start_us":147,"threads_started":1}
I20260620 18:39:30.997740 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.982833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:59717 (local address 127.10.250.126:46691)
0620 18:39:30.982930 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.982932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.983144 (+ 212us) server_negotiation.cc:407] Connection header received
0620 18:39:30.983630 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.983632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.983655 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.983686 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.984800 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.985585 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.987133 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.987444 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.987486 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.993048 (+ 5562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.993049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.993050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.993058 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.996305 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.996842 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.996844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.996844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.996902 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.997106 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.997107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.997107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.997270 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.997327 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.997552 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.997629 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:39:30.997721 18583 heartbeater.cc:347] Connected to a master server at 127.10.250.125:35565
I20260620 18:39:30.997828 18583 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:30.998054 18583 heartbeater.cc:511] Master 127.10.250.125:35565 requested a full tablet report, sending...
I20260620 18:39:30.998474 18587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.982468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.67:59717)
0620 18:39:30.983055 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.983065 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.983509 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.983772 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.983773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.984340 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.984681 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.984688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.985703 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.985704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.987029 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.987032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.987580 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.987581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.987736 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.988588 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.988594 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.992922 (+ 4328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:30.996490 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.996491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.996492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.996725 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.996972 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:30.996973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:30.996973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:30.997035 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:30.997381 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:30.997382 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:30.997427 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:39:30.997882 (+ 455us) client_negotiation.cc:240] Negotiation successful
0620 18:39:30.998002 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":503,"thread_start_us":104,"threads_started":1}
I20260620 18:39:30.998678 17943 ts_manager.cc:194] Registered new tserver with Master: 5d3a1b3e7d4441c7ad394c3aa6409cc4 (127.10.250.67:36941)
I20260620 18:39:30.998790 18582 heartbeater.cc:347] Connected to a master server at 127.10.250.126:46691
I20260620 18:39:30.998840 18582 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:30.998908 18582 heartbeater.cc:511] Master 127.10.250.126:46691 requested a full tablet report, sending...
I20260620 18:39:30.999115 17345 ts_manager.cc:194] Re-registered known tserver with Master: 5d3a1b3e7d4441c7ad394c3aa6409cc4 (127.10.250.67:36941)
I20260620 18:39:30.999610 17345 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:59717
I20260620 18:39:30.999650 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.986999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55472 (local address 127.10.250.124:35477)
0620 18:39:30.989029 (+ 2030us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.989031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.989038 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:39:30.989059 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.989060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.989076 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.989157 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.989819 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.990627 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.991821 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.992143 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.992207 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.995258 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.995259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.995260 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.995267 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.997759 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.998329 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.998330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.998330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.998394 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.998761 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.998762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.998763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.998949 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.999039 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.999452 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.999540 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2013,"thread_start_us":61,"threads_started":1}
I20260620 18:39:31.000422 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.988842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:35477 (local address 127.0.0.1:55478)
0620 18:39:30.988880 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.988891 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.988930 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.991020 (+ 2090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.991021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.991023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:30.991066 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.991067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.991975 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.991977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.995280 (+ 3303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.995283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.000188 (+ 4905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.000189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:31.000255 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.000259 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:39:31.000290 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 18:39:31.000335 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 18:39:31.001080 18596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.988903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.10.250.124:35477)
0620 18:39:30.990566 (+ 1663us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.990568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.990575 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:39:30.990596 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.990597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.990634 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.991137 (+ 503us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:30.991175 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.991872 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.998277 (+ 6405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.998956 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.000840 (+ 1884us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.000930 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:31.000949 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 18:39:31.001003 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1643,"thread_start_us":88,"threads_started":1}
I20260620 18:39:31.001439 18139 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 18:39:31.001639 18592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.986089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.10.250.126:46691)
0620 18:39:30.987335 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.987337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.988098 (+ 761us) server_negotiation.cc:407] Connection header received
0620 18:39:30.988141 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.988142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.988176 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.988248 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.989340 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.990145 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.991366 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.991752 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.991810 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.993954 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.993955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.993955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.993963 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.998163 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.999342 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.999343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.999343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.999494 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:31.000057 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:31.000058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:31.000058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:31.000676 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:31.000784 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:31.001435 (+ 651us) server_negotiation.cc:299] Negotiation successful
0620 18:39:31.001526 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1223,"thread_start_us":90,"threads_started":1}
I20260620 18:39:31.001665 18168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91b96f5e5c59435abc16847a2b3db027. Latest consensus state: current_term: 1 leader_uuid: "91b96f5e5c59435abc16847a2b3db027" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } } }
I20260620 18:39:31.001760 18168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:39:31.003466 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.982887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:54901 (local address 127.10.250.124:35477)
0620 18:39:30.982964 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.982966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.982975 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:30.983578 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.983579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.983591 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.983628 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.984809 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.985564 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.987135 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.987440 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.987486 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.000427 (+ 12941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:31.000428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:31.000429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:31.000436 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:31.002113 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:31.002540 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:31.002540 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:31.002541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:31.002600 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:31.002852 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:31.002853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:31.002853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:31.003000 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:31.003073 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:31.003289 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 18:39:31.003374 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:39:31.003490 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.982468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:35477 (local address 127.10.250.67:54901)
0620 18:39:30.982712 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:30.982755 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:30.983470 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:30.983705 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:30.983709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:30.984492 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:30.984683 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.984691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.985687 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.985688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.987027 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:30.987029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.987578 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.987578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:30.987736 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.988595 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:30.988599 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:30.998345 (+ 9746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:31.002219 (+ 3874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:31.002220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:31.002221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:31.002451 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:31.002694 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:31.002695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:31.002695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:31.002767 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:31.003102 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:31.003105 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:31.003157 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 18:39:31.003232 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 18:39:31.003360 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":50,"spinlock_wait_cycles":98560,"thread_start_us":166,"threads_started":1}
I20260620 18:39:31.003854 18581 heartbeater.cc:347] Connected to a master server at 127.10.250.124:35477
I20260620 18:39:31.003925 18581 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:31.004024 18581 heartbeater.cc:511] Master 127.10.250.124:35477 requested a full tablet report, sending...
I20260620 18:39:31.004446 18129 ts_manager.cc:194] Registered new tserver with Master: 5d3a1b3e7d4441c7ad394c3aa6409cc4 (127.10.250.67:36941)
I20260620 18:39:31.004714 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.986475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.10.250.125:35565)
0620 18:39:30.987280 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.987282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.987497 (+ 215us) server_negotiation.cc:407] Connection header received
0620 18:39:30.987522 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.987522 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.987536 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.987593 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.989671 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.990432 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.993193 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.993576 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.993649 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.996516 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.996518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.996518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.996525 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:31.002478 (+ 5953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:31.002882 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:31.002883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:31.002884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:31.002943 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:31.003188 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:31.003189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:31.003190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:31.003494 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:31.004268 (+ 774us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:31.004516 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 18:39:31.004607 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":80,"threads_started":1}
I20260620 18:39:31.004464 17345 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:31.006271 17345 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:31.007159 18090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91b96f5e5c59435abc16847a2b3db027" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: NON_VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: true } } }
I20260620 18:39:31.007344 18090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:31.009059 17904 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027: attempting to promote NON_VOTER 834ae29701f044b2a8e99747efffa486 to VOTER
I20260620 18:39:31.009208 17904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } }
I20260620 18:39:31.009958 18139 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 1 LEARNER]: Refusing update from remote peer 91b96f5e5c59435abc16847a2b3db027: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 18:39:31.009995 17953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 FOLLOWER]: Refusing update from remote peer 91b96f5e5c59435abc16847a2b3db027: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 18:39:31.010160 17905 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 18:39:31.010251 17904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 18:39:31.011124 17905 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 834ae29701f044b2a8e99747efffa486 (127.10.250.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } } }
I20260620 18:39:31.011270 17953 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 834ae29701f044b2a8e99747efffa486 (127.10.250.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } } }
I20260620 18:39:31.012082 17905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "91b96f5e5c59435abc16847a2b3db027" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } } }
I20260620 18:39:31.012192 17905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:31.012267 17905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91b96f5e5c59435abc16847a2b3db027" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } } }
I20260620 18:39:31.012333 17905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:31.012301 18090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91b96f5e5c59435abc16847a2b3db027" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } } }
I20260620 18:39:31.012212 18139 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 834ae29701f044b2a8e99747efffa486 (127.10.250.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } } }
I20260620 18:39:31.012375 18090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:31.012410 17982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91b96f5e5c59435abc16847a2b3db027" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } } }
I20260620 18:39:31.012490 17982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:39:31.013187 18168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91b96f5e5c59435abc16847a2b3db027" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } } }
I20260620 18:39:31.013286 18168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:39:31.017608 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.013605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:36941 (local address 127.0.0.1:44420)
0620 18:39:31.013770 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:31.013779 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:31.013819 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:31.014258 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:31.014259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:31.014260 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:31.014322 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:31.014323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.015734 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.015735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:31.016488 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:31.016490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.017351 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.017351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:31.017434 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.017440 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:39:31.017503 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 18:39:31.017546 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":5,"thread_start_us":118,"threads_started":1}
I20260620 18:39:31.017640 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.013869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44420 (local address 127.10.250.67:36941)
0620 18:39:31.014070 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:31.014075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:31.014086 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:39:31.014129 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:31.014132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:31.014148 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:31.014398 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:31.014455 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.015633 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.016583 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.017278 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.017353 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.017411 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:31.017518 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 18:39:31.017557 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":98,"threads_started":1}
I20260620 18:39:31.018392 18515 tablet_service.cc:1467] Processing CreateTablet for tablet ad571694606c4fec81d258fff284ac52 (DEFAULT_TABLE table=test-table [id=b47356463910430c8b858f55a645dd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:31.018838 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.013557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:33437 (local address 127.0.0.1:37832)
0620 18:39:31.013600 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:31.013613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:31.013652 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:31.014258 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:31.014259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:31.014260 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:31.014322 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:31.014323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.015683 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.015684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:31.016890 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:31.016893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.018624 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.018625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:31.018691 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.018694 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:31.018726 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 18:39:31.018769 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":42}
I20260620 18:39:31.018863 18515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad571694606c4fec81d258fff284ac52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:31.019102 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.013635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:37457 (local address 127.0.0.1:41502)
0620 18:39:31.013896 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:31.013902 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:31.013944 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:31.014411 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:31.014412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:31.014414 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:31.014465 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:31.014466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.015593 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.015594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:31.016901 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:31.016903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.018672 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.018674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:31.018728 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.018732 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:39:31.018773 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 18:39:31.018804 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":91,"thread_start_us":100,"threads_started":1}
I20260620 18:39:31.019222 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.013651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.10.250.65:33437)
0620 18:39:31.014070 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:31.014075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:31.014086 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:39:31.014129 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:31.014132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:31.014148 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:31.014398 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:31.014456 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.015610 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.016976 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.017696 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.018968 (+ 1272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.019060 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:31.019087 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 18:39:31.019133 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":100,"threads_started":1}
I20260620 18:39:31.019235 18611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.014037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.10.250.66:37457)
0620 18:39:31.014201 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:31.014205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:31.014218 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:39:31.014262 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:31.014264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:31.014282 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:31.014407 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:31.014566 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.015460 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.017786 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.018484 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.018615 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.019073 (+ 458us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:31.019104 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 18:39:31.019148 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260620 18:39:31.020013 18379 tablet_service.cc:1467] Processing CreateTablet for tablet ad571694606c4fec81d258fff284ac52 (DEFAULT_TABLE table=test-table [id=b47356463910430c8b858f55a645dd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:31.020380 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad571694606c4fec81d258fff284ac52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:31.021883 18612 tablet_bootstrap.cc:492] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: Bootstrap starting.
I20260620 18:39:31.022591 18612 tablet_bootstrap.cc:654] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:31.022459 18242 tablet_service.cc:1467] Processing CreateTablet for tablet ad571694606c4fec81d258fff284ac52 (DEFAULT_TABLE table=test-table [id=b47356463910430c8b858f55a645dd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:31.022866 18242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad571694606c4fec81d258fff284ac52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:31.022982 18612 log.cc:824] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:31.023865 18612 tablet_bootstrap.cc:492] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: No bootstrap required, opened a new log
I20260620 18:39:31.023932 18612 ts_tablet_manager.cc:1397] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:39:31.026397 18615 tablet_bootstrap.cc:492] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c: Bootstrap starting.
I20260620 18:39:31.026544 18612 raft_consensus.cc:348] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:31.026707 18612 raft_consensus.cc:374] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:31.026731 18612 raft_consensus.cc:729] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3a1b3e7d4441c7ad394c3aa6409cc4, State: Initialized, Role: FOLLOWER
I20260620 18:39:31.026870 18612 consensus_queue.cc:260] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:31.027128 18615 tablet_bootstrap.cc:654] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:31.027136 18612 ts_tablet_manager.cc:1428] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 18:39:31.027283 18582 heartbeater.cc:503] Master 127.10.250.126:46691 was elected leader, sending a full tablet report...
I20260620 18:39:31.027482 18615 log.cc:824] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:31.027807 18614 tablet_bootstrap.cc:492] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88: Bootstrap starting.
I20260620 18:39:31.028551 18615 tablet_bootstrap.cc:492] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c: No bootstrap required, opened a new log
I20260620 18:39:31.028635 18615 ts_tablet_manager.cc:1397] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:39:31.028625 18614 tablet_bootstrap.cc:654] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:31.028909 18614 log.cc:824] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:31.029708 18614 tablet_bootstrap.cc:492] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88: No bootstrap required, opened a new log
I20260620 18:39:31.029793 18614 ts_tablet_manager.cc:1397] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 18:39:31.030609 18615 raft_consensus.cc:348] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:31.030753 18615 raft_consensus.cc:374] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:31.030795 18615 raft_consensus.cc:729] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ef94b55fa94b9c90a32215dfb2fa2c, State: Initialized, Role: FOLLOWER
I20260620 18:39:31.030916 18615 consensus_queue.cc:260] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:31.031176 18615 ts_tablet_manager.cc:1428] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 18:39:31.031275 18309 heartbeater.cc:503] Master 127.10.250.126:46691 was elected leader, sending a full tablet report...
I20260620 18:39:31.031534 18614 raft_consensus.cc:348] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:31.031689 18614 raft_consensus.cc:374] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:31.031734 18614 raft_consensus.cc:729] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4e832d399164153a9c69cae14100b88, State: Initialized, Role: FOLLOWER
I20260620 18:39:31.031852 18614 consensus_queue.cc:260] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:31.032125 18614 ts_tablet_manager.cc:1428] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 18:39:31.032269 18446 heartbeater.cc:503] Master 127.10.250.126:46691 was elected leader, sending a full tablet report...
W20260620 18:39:31.097585 18448 tablet.cc:2369] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:39:31.100541 18620 raft_consensus.cc:482] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:31.100765 18620 raft_consensus.cc:504] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:31.101202 18620 leader_election.cc:288] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4ef94b55fa94b9c90a32215dfb2fa2c (127.10.250.65:33437), 5d3a1b3e7d4441c7ad394c3aa6409cc4 (127.10.250.67:36941)
I20260620 18:39:31.104319 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.101448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:36941 (local address 127.10.250.66:39511)
0620 18:39:31.101530 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:31.101542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:31.101594 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:31.101790 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:31.101791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:31.101793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:31.101872 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:31.101874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.102633 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.102633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:31.103392 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:31.103393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.104041 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.104041 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:31.104119 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.104123 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:39:31.104174 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 18:39:31.104236 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 18:39:31.104328 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.101484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:33437 (local address 127.10.250.66:55481)
0620 18:39:31.101672 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:31.101681 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:31.101726 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:31.101956 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:31.101957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:31.101959 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:31.102029 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:31.102031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.102756 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.102757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:31.103463 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:31.103465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.104099 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.104099 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:31.104168 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.104171 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:31.104217 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 18:39:31.104247 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260620 18:39:31.104483 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.101508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:55481 (local address 127.10.250.65:33437)
0620 18:39:31.101555 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:31.101558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:31.101758 (+ 200us) server_negotiation.cc:407] Connection header received
0620 18:39:31.101790 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:31.101791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:31.101801 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:31.101870 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:31.102110 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.102670 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.103561 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.104029 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.104099 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.104108 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:31.104358 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 18:39:31.104403 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:39:31.104607 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.101511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:39511 (local address 127.10.250.67:36941)
0620 18:39:31.101569 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:31.101571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:31.101582 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:39:31.101663 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:31.101664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:31.101686 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:31.101752 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:31.101937 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.102482 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.103498 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.103974 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.104044 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.104053 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:31.104499 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 18:39:31.104543 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:39:31.104933 18535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad571694606c4fec81d258fff284ac52" candidate_uuid: "e4e832d399164153a9c69cae14100b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" is_pre_election: true
I20260620 18:39:31.104933 18262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad571694606c4fec81d258fff284ac52" candidate_uuid: "e4e832d399164153a9c69cae14100b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" is_pre_election: true
I20260620 18:39:31.105100 18535 raft_consensus.cc:2454] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4e832d399164153a9c69cae14100b88 in term 0.
I20260620 18:39:31.105118 18262 raft_consensus.cc:2454] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4e832d399164153a9c69cae14100b88 in term 0.
I20260620 18:39:31.105338 18335 leader_election.cc:302] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3a1b3e7d4441c7ad394c3aa6409cc4, e4e832d399164153a9c69cae14100b88; no voters:
I20260620 18:39:31.105484 18620 raft_consensus.cc:2781] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:31.105537 18620 raft_consensus.cc:482] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:31.105561 18620 raft_consensus.cc:3037] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:31.106274 18620 raft_consensus.cc:504] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:31.106420 18620 leader_election.cc:288] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [CANDIDATE]: Term 1 election: Requested vote from peers c4ef94b55fa94b9c90a32215dfb2fa2c (127.10.250.65:33437), 5d3a1b3e7d4441c7ad394c3aa6409cc4 (127.10.250.67:36941)
I20260620 18:39:31.106576 18535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad571694606c4fec81d258fff284ac52" candidate_uuid: "e4e832d399164153a9c69cae14100b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4"
I20260620 18:39:31.106663 18535 raft_consensus.cc:3037] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:31.107365 18535 raft_consensus.cc:2454] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4e832d399164153a9c69cae14100b88 in term 1.
I20260620 18:39:31.107486 18262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad571694606c4fec81d258fff284ac52" candidate_uuid: "e4e832d399164153a9c69cae14100b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c"
I20260620 18:39:31.107537 18335 leader_election.cc:302] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3a1b3e7d4441c7ad394c3aa6409cc4, e4e832d399164153a9c69cae14100b88; no voters:
I20260620 18:39:31.107566 18262 raft_consensus.cc:3037] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:31.107632 18620 raft_consensus.cc:2781] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:31.107811 18620 raft_consensus.cc:686] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [term 1 LEADER]: Becoming Leader. State: Replica: e4e832d399164153a9c69cae14100b88, State: Running, Role: LEADER
I20260620 18:39:31.107910 18620 consensus_queue.cc:237] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:31.108472 18262 raft_consensus.cc:2454] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4e832d399164153a9c69cae14100b88 in term 1.
I20260620 18:39:31.108899 17345 catalog_manager.cc:5495] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4e832d399164153a9c69cae14100b88 (127.10.250.66). New cstate: current_term: 1 leader_uuid: "e4e832d399164153a9c69cae14100b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } health_report { overall_health: HEALTHY } } }
I20260620 18:39:31.132195 18611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.129621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.10.250.66:37457)
0620 18:39:31.129687 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:31.129689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:31.129702 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:39:31.129736 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:31.129736 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:31.129755 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:31.129826 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:31.129991 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.130472 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.131433 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.131628 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.131695 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.131841 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:31.131956 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:31.132022 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:31.132057 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 18:39:31.132076 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 18:39:31.134084 18262 raft_consensus.cc:1261] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c [term 1 FOLLOWER]: Refusing update from remote peer e4e832d399164153a9c69cae14100b88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:31.134083 18535 raft_consensus.cc:1261] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 1 FOLLOWER]: Refusing update from remote peer e4e832d399164153a9c69cae14100b88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:31.134485 18620 consensus_queue.cc:1035] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:31.134696 18621 consensus_queue.cc:1035] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:31.137468 18624 mvcc.cc:205] Tried to move back new op lower bound from 7298993238562041856 to 7298993238458859520. Current Snapshot: MvccSnapshot[applied={T|T < 7298993238562041856 or (T in {7298993238562041856})}]
I20260620 18:39:31.140069 18626 mvcc.cc:205] Tried to move back new op lower bound from 7298993238562041856 to 7298993238458859520. Current Snapshot: MvccSnapshot[applied={T|T < 7298993238562041856}]
I20260620 18:39:31.140219 17344 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:31.140323 17344 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:31.141849 17344 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91b96f5e5c59435abc16847a2b3db027: Sending DeleteTablet for 3 replicas of tablet ad571694606c4fec81d258fff284ac52
I20260620 18:39:31.142151 18242 tablet_service.cc:1514] Processing DeleteTablet for tablet ad571694606c4fec81d258fff284ac52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:31 UTC) from {username='oryx', principal='oryx/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:37832
I20260620 18:39:31.142175 18379 tablet_service.cc:1514] Processing DeleteTablet for tablet ad571694606c4fec81d258fff284ac52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:31 UTC) from {username='oryx', principal='oryx/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:41502
I20260620 18:39:31.142266 18515 tablet_service.cc:1514] Processing DeleteTablet for tablet ad571694606c4fec81d258fff284ac52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:31 UTC) from {username='oryx', principal='oryx/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:44420
I20260620 18:39:31.142439 18635 tablet_replica.cc:330] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88: stopping tablet replica
I20260620 18:39:31.142542 18636 tablet_replica.cc:330] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: stopping tablet replica
I20260620 18:39:31.142573 18635 raft_consensus.cc:2229] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:31.142673 18636 raft_consensus.cc:2229] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:31.142684 18635 raft_consensus.cc:2258] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:31.142793 18636 raft_consensus.cc:2258] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:31.143148 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 17313
I20260620 18:39:31.143908 18636 ts_tablet_manager.cc:1905] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:31.145846 18636 ts_tablet_manager.cc:1918] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:31.145946 18636 log.cc:1196] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/ad571694606c4fec81d258fff284ac52
I20260620 18:39:31.146224 18636 ts_tablet_manager.cc:1939] T ad571694606c4fec81d258fff284ac52 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: Deleting consensus metadata
I20260620 18:39:31.147549 18635 ts_tablet_manager.cc:1905] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 18:39:31.149034 18471 connection.cc:535] server connection from 127.0.0.1:44420 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 18:39:31.149412 18635 ts_tablet_manager.cc:1918] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:31.149488 18635 log.cc:1196] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/ad571694606c4fec81d258fff284ac52
I20260620 18:39:31.149786 18635 ts_tablet_manager.cc:1939] T ad571694606c4fec81d258fff284ac52 P e4e832d399164153a9c69cae14100b88: Deleting consensus metadata
W20260620 18:39:31.150175 18332 connection.cc:441] server connection from 127.0.0.1:41502 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41502 (request call id 1) could send its response
I20260620 18:39:31.152408 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.143034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.10.250.124:35477)
0620 18:39:31.143077 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:31.143079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:31.144253 (+ 1174us) server_negotiation.cc:407] Connection header received
0620 18:39:31.144279 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:31.144280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:31.144291 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:31.144353 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:31.145107 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.145852 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.146918 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.147157 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.147222 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.149079 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:31.149080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:31.149081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:31.149088 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:31.151014 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:31.151433 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:31.151435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:31.151435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:31.151520 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:31.151751 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:31.151752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:31.151753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:31.151893 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:31.151971 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:31.152190 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 18:39:31.152277 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:39:31.154717 18634 tablet_replica.cc:330] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c: stopping tablet replica
I20260620 18:39:31.154862 18634 raft_consensus.cc:2229] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:31.154999 18634 raft_consensus.cc:2258] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:31.156199 18634 ts_tablet_manager.cc:1905] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:31.159168 18634 ts_tablet_manager.cc:1918] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:31.159250 18634 log.cc:1196] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-0/wal/wals/ad571694606c4fec81d258fff284ac52
I20260620 18:39:31.159490 18596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.151829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55492 (local address 127.10.250.124:35477)
0620 18:39:31.151871 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:31.151873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:31.151887 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:39:31.151905 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:31.151906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:31.151917 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:31.151960 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:31.152552 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.153310 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.154328 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.154667 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.154736 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.156292 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:31.156293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:31.156293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:31.156301 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:31.157938 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:31.158342 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:31.158343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:31.158344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:31.158399 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:31.158625 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:31.158626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:31.158627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:31.158754 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:31.158848 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:31.159251 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 18:39:31.159394 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:39:31.159648 18634 ts_tablet_manager.cc:1939] T ad571694606c4fec81d258fff284ac52 P c4ef94b55fa94b9c90a32215dfb2fa2c: Deleting consensus metadata
I20260620 18:39:31.160223 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.151971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.10.250.125:35565)
0620 18:39:31.152013 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:31.152015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:31.152640 (+ 625us) server_negotiation.cc:407] Connection header received
0620 18:39:31.152734 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:31.152735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:31.152749 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:31.152779 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:31.153351 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.154112 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.155116 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.155430 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.155475 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.156988 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:31.156989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:31.156989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:31.156996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:31.158720 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:31.159160 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:31.159161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:31.159162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:31.159217 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:31.159502 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:31.159503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:31.159503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:31.159635 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:31.159707 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:31.160042 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 18:39:31.160123 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 18:39:31.160214 18195 connection.cc:441] server connection from 127.0.0.1:37832 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37832 (request call id 1) could send its response
I20260620 18:39:32.138466 18644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.138183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.65:56581)
0620 18:39:32.138327 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.138408 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
W20260620 18:39:32.138767 18309 heartbeater.cc:644] Failed to heartbeat to 127.10.250.126:46691 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
I20260620 18:39:32.139551 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.139336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.66:36229)
0620 18:39:32.139465 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.139517 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":74,"threads_started":1}
W20260620 18:39:32.139835 18446 heartbeater.cc:644] Failed to heartbeat to 127.10.250.126:46691 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
I20260620 18:39:32.140300 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.140057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.67:51651)
0620 18:39:32.140183 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.140251 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":49,"threads_started":1}
W20260620 18:39:32.140516 18582 heartbeater.cc:644] Failed to heartbeat to 127.10.250.126:46691 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
I20260620 18:39:32.642320 18647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 91b96f5e5c59435abc16847a2b3db027)
I20260620 18:39:32.642344 18648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 91b96f5e5c59435abc16847a2b3db027)
I20260620 18:39:32.642454 18647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } }
I20260620 18:39:32.642546 18648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } }
I20260620 18:39:32.642835 18647 leader_election.cc:288] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91b96f5e5c59435abc16847a2b3db027 (127.10.250.126:46691), 7d2401dd5ea84376b1824c16cdc174ae (127.10.250.125:35565)
I20260620 18:39:32.643064 18648 leader_election.cc:288] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91b96f5e5c59435abc16847a2b3db027 (127.10.250.126:46691), 834ae29701f044b2a8e99747efffa486 (127.10.250.124:35477)
I20260620 18:39:32.643188 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.642955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34598)
0620 18:39:32.643073 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.643126 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
W20260620 18:39:32.643316 17929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
I20260620 18:39:32.643530 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.642932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34592)
0620 18:39:32.643176 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.643224 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":71,"threads_started":1}
W20260620 18:39:32.643625 18115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
I20260620 18:39:32.643952 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.643621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34606)
0620 18:39:32.643877 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.643916 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238}
I20260620 18:39:32.643998 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.643919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34608)
0620 18:39:32.643954 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.643975 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 18:39:32.644073 18115 leader_election.cc:334] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 91b96f5e5c59435abc16847a2b3db027 (127.10.250.126:46691): Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
W20260620 18:39:32.644181 17929 leader_election.cc:334] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 91b96f5e5c59435abc16847a2b3db027 (127.10.250.126:46691): Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
I20260620 18:39:32.647195 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.642988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:35565 (local address 127.0.0.1:57034)
0620 18:39:32.643403 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.643412 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:32.643468 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:32.643672 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:32.643672 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:32.643674 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:32.643720 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:32.643722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.644699 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.644701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:32.645976 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:32.645979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.646848 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.646849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:32.646925 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.646930 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:39:32.647013 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 18:39:32.647061 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":43,"threads_started":1}
I20260620 18:39:32.647222 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.643084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57034 (local address 127.10.250.125:35565)
0620 18:39:32.643180 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:32.643183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:32.643521 (+ 338us) server_negotiation.cc:407] Connection header received
0620 18:39:32.643548 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:32.643548 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:32.643558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:32.643624 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:32.643792 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.644583 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.646097 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.646726 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.646821 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.646830 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:32.647051 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 18:39:32.647102 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":47,"threads_started":1}
I20260620 18:39:32.647209 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.643373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.10.250.124:35477)
0620 18:39:32.643514 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:32.643516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:32.643535 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:39:32.643553 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:32.643554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:32.643562 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:32.643624 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:32.643792 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.644583 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.646098 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.646732 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.646821 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.646830 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:32.647052 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 18:39:32.647102 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":43,"threads_started":1}
I20260620 18:39:32.647459 17953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "834ae29701f044b2a8e99747efffa486" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7d2401dd5ea84376b1824c16cdc174ae" is_pre_election: true
I20260620 18:39:32.647527 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.643221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:35477 (local address 127.0.0.1:55500)
0620 18:39:32.643255 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.643264 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:32.643323 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:32.643672 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:32.643672 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:32.643674 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:32.643720 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:32.643722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.644704 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.644705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:32.645976 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:32.645979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.646848 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.646849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:32.646925 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.646931 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:39:32.647013 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 18:39:32.647060 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"thread_start_us":55,"threads_started":1}
I20260620 18:39:32.647576 17953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 834ae29701f044b2a8e99747efffa486 in term 1.
I20260620 18:39:32.647809 18116 leader_election.cc:302] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d2401dd5ea84376b1824c16cdc174ae, 834ae29701f044b2a8e99747efffa486; no voters: 91b96f5e5c59435abc16847a2b3db027
I20260620 18:39:32.647945 18139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d2401dd5ea84376b1824c16cdc174ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "834ae29701f044b2a8e99747efffa486" is_pre_election: true
I20260620 18:39:32.648044 18139 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d2401dd5ea84376b1824c16cdc174ae in term 1.
I20260620 18:39:32.648128 18647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 18:39:32.648168 18647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 91b96f5e5c59435abc16847a2b3db027)
I20260620 18:39:32.648190 18647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:39:32.648212 17927 leader_election.cc:302] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d2401dd5ea84376b1824c16cdc174ae, 834ae29701f044b2a8e99747efffa486; no voters: 91b96f5e5c59435abc16847a2b3db027
I20260620 18:39:32.648317 18648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 18:39:32.648365 18648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 FOLLOWER]: Starting leader election (detected failure of leader 91b96f5e5c59435abc16847a2b3db027)
I20260620 18:39:32.648391 18648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:39:32.649045 18647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } }
I20260620 18:39:32.649081 18648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } }
I20260620 18:39:32.649200 18647 leader_election.cc:288] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [CANDIDATE]: Term 2 election: Requested vote from peers 91b96f5e5c59435abc16847a2b3db027 (127.10.250.126:46691), 7d2401dd5ea84376b1824c16cdc174ae (127.10.250.125:35565)
I20260620 18:39:32.649235 18648 leader_election.cc:288] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [CANDIDATE]: Term 2 election: Requested vote from peers 91b96f5e5c59435abc16847a2b3db027 (127.10.250.126:46691), 834ae29701f044b2a8e99747efffa486 (127.10.250.124:35477)
I20260620 18:39:32.649441 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.649281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34620)
0620 18:39:32.649386 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.649415 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 18:39:32.649430 17953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "834ae29701f044b2a8e99747efffa486" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7d2401dd5ea84376b1824c16cdc174ae"
I20260620 18:39:32.649492 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.649371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34632)
0620 18:39:32.649411 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.649439 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 18:39:32.649567 17953 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 834ae29701f044b2a8e99747efffa486 in current term 2: Already voted for candidate 7d2401dd5ea84376b1824c16cdc174ae in this term.
I20260620 18:39:32.649664 18139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d2401dd5ea84376b1824c16cdc174ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "834ae29701f044b2a8e99747efffa486"
I20260620 18:39:32.649739 18139 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d2401dd5ea84376b1824c16cdc174ae in current term 2: Already voted for candidate 834ae29701f044b2a8e99747efffa486 in this term.
I20260620 18:39:32.649874 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.649779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34642)
0620 18:39:32.649824 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.649850 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 18:39:32.649874 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.649795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34652)
0620 18:39:32.649830 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.649850 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 18:39:32.649940 18115 leader_election.cc:334] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 91b96f5e5c59435abc16847a2b3db027 (127.10.250.126:46691): Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
W20260620 18:39:32.649952 17929 leader_election.cc:334] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 91b96f5e5c59435abc16847a2b3db027 (127.10.250.126:46691): Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
I20260620 18:39:32.649982 18115 leader_election.cc:302] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 834ae29701f044b2a8e99747efffa486; no voters: 7d2401dd5ea84376b1824c16cdc174ae, 91b96f5e5c59435abc16847a2b3db027
I20260620 18:39:32.649991 17929 leader_election.cc:302] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d2401dd5ea84376b1824c16cdc174ae; no voters: 834ae29701f044b2a8e99747efffa486, 91b96f5e5c59435abc16847a2b3db027
I20260620 18:39:32.650076 18648 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 18:39:32.650156 18647 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 18:39:33.139376 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:33.139143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.65:59351)
0620 18:39:33.139270 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:33.139315 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260620 18:39:33.140450 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:33.140204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.66:37889)
0620 18:39:33.140357 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:33.140403 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260620 18:39:33.140985 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:33.140794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.67:36577)
0620 18:39:33.140882 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:33.140940 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":50,"threads_started":1}
I20260620 18:39:34.140235 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.139953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.65:43709)
0620 18:39:34.140067 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.140135 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
I20260620 18:39:34.141178 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.140965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.66:44813)
0620 18:39:34.141092 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.141129 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":54,"threads_started":1}
I20260620 18:39:34.141539 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.141353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.67:49517)
0620 18:39:34.141466 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.141498 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":45,"threads_started":1}
I20260620 18:39:34.356076 18663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:34.356166 18663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } }
I20260620 18:39:34.356344 18663 leader_election.cc:288] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 91b96f5e5c59435abc16847a2b3db027 (127.10.250.126:46691), 7d2401dd5ea84376b1824c16cdc174ae (127.10.250.125:35565)
I20260620 18:39:34.356644 17953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "834ae29701f044b2a8e99747efffa486" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7d2401dd5ea84376b1824c16cdc174ae" is_pre_election: true
I20260620 18:39:34.356734 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.356502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34880)
0620 18:39:34.356642 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.356692 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260620 18:39:34.356763 17953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 834ae29701f044b2a8e99747efffa486 in term 2.
I20260620 18:39:34.356917 18116 leader_election.cc:302] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d2401dd5ea84376b1824c16cdc174ae, 834ae29701f044b2a8e99747efffa486; no voters:
I20260620 18:39:34.357031 18663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 18:39:34.357069 18663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:34.357122 18663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:39:34.357174 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.357078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34882)
0620 18:39:34.357122 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.357147 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 18:39:34.357242 18115 leader_election.cc:334] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 91b96f5e5c59435abc16847a2b3db027 (127.10.250.126:46691): Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
I20260620 18:39:34.357916 18663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } }
I20260620 18:39:34.358062 18663 leader_election.cc:288] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [CANDIDATE]: Term 3 election: Requested vote from peers 91b96f5e5c59435abc16847a2b3db027 (127.10.250.126:46691), 7d2401dd5ea84376b1824c16cdc174ae (127.10.250.125:35565)
I20260620 18:39:34.358212 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.358134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34898)
0620 18:39:34.358167 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.358190 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 18:39:34.358215 17953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "834ae29701f044b2a8e99747efffa486" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7d2401dd5ea84376b1824c16cdc174ae"
I20260620 18:39:34.358310 17953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:39:34.358611 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.358478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34908)
0620 18:39:34.358558 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.358585 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 18:39:34.358711 18115 leader_election.cc:334] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 91b96f5e5c59435abc16847a2b3db027 (127.10.250.126:46691): Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
I20260620 18:39:34.359134 17953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 834ae29701f044b2a8e99747efffa486 in term 3.
I20260620 18:39:34.359294 18116 leader_election.cc:302] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d2401dd5ea84376b1824c16cdc174ae, 834ae29701f044b2a8e99747efffa486; no voters: 91b96f5e5c59435abc16847a2b3db027
I20260620 18:39:34.359436 18663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:39:34.359601 18663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [term 3 LEADER]: Becoming Leader. State: Replica: 834ae29701f044b2a8e99747efffa486, State: Running, Role: LEADER
I20260620 18:39:34.359694 18663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } }
I20260620 18:39:34.360085 18666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 834ae29701f044b2a8e99747efffa486. Latest consensus state: current_term: 3 leader_uuid: "834ae29701f044b2a8e99747efffa486" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } } }
I20260620 18:39:34.360168 18666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:34.360303 18668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:34.360718 18668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:34.360813 18668 catalog_manager.cc:1259] Loaded cluster ID: a2dd3d882d904832ba1e2c8e352f2c66
I20260620 18:39:34.360845 18668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:34.361021 18668 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:34.361112 18668 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: Loaded TSK: 0
I20260620 18:39:34.361280 18668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:39:34.401418 18129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:34.403227 18129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:34.404379 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.404252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34920)
0620 18:39:34.404312 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.404349 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 18:39:34.404374 17953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [term 3 FOLLOWER]: Refusing update from remote peer 834ae29701f044b2a8e99747efffa486: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 18:39:34.404608 18663 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 18:39:34.404762 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.404657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.0.0.1:34932)
0620 18:39:34.404705 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.404731 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 18:39:34.404860 18115 consensus_peers.cc:487] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 -> Peer 91b96f5e5c59435abc16847a2b3db027 (127.10.250.126:46691): Couldn't send request to peer 91b96f5e5c59435abc16847a2b3db027. Status: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 18:39:34.405355 18670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 834ae29701f044b2a8e99747efffa486. Latest consensus state: current_term: 3 leader_uuid: "834ae29701f044b2a8e99747efffa486" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } } }
I20260620 18:39:34.405478 18670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:39:34.405717 18669 mvcc.cc:205] Tried to move back new op lower bound from 7298993251958063104 to 7298993251778043904. Current Snapshot: MvccSnapshot[applied={T|T < 7298993251958063104}]
I20260620 18:39:34.405828 18663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "834ae29701f044b2a8e99747efffa486" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } } }
I20260620 18:39:34.405911 18663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:34.406045 18671 mvcc.cc:205] Tried to move back new op lower bound from 7298993251958063104 to 7298993251778043904. Current Snapshot: MvccSnapshot[applied={T|T < 7298993251958063104}]
I20260620 18:39:34.406145 18670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "834ae29701f044b2a8e99747efffa486" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "91b96f5e5c59435abc16847a2b3db027" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 46691 } } peers { permanent_uuid: "7d2401dd5ea84376b1824c16cdc174ae" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 35565 } attrs { promote: false } } peers { permanent_uuid: "834ae29701f044b2a8e99747efffa486" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 35477 } attrs { promote: false } } }
I20260620 18:39:34.406225 18670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2401dd5ea84376b1824c16cdc174ae [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:39:34.411799 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.408148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.10.250.67:36941)
0620 18:39:34.408390 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:34.408392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:34.408400 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:39:34.408424 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:34.408424 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:34.408434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:34.408501 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:34.408620 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:34.409457 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:34.410824 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:34.411385 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:34.411475 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:34.411487 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:34.411634 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 18:39:34.411678 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":56,"threads_started":1}
I20260620 18:39:34.411794 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.407884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:33437 (local address 127.0.0.1:37842)
0620 18:39:34.407917 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.407930 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:34.407981 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:34.408304 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:34.408306 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:34.408318 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:34.408387 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:34.408389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:34.409450 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:34.409451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:34.410680 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:34.410683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:34.411472 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:34.411473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:34.411541 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:34.411546 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:39:34.411582 (+ 36us) client_negotiation.cc:240] Negotiation successful
0620 18:39:34.411661 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":5}
I20260620 18:39:34.411794 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.407970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:37457 (local address 127.0.0.1:41524)
0620 18:39:34.408233 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.408242 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:34.408283 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:34.408713 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:34.408714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:34.408716 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:34.408837 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:34.408840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:34.409670 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:34.409671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:34.410792 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:34.410795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:34.411507 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:34.411507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:34.411556 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:34.411559 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:34.411615 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 18:39:34.411670 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":7,"thread_start_us":82,"threads_started":1}
I20260620 18:39:34.411964 18678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.408003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.10.250.65:33437)
0620 18:39:34.408141 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:34.408144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:34.408152 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:39:34.408182 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:34.408183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:34.408195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:34.408457 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:34.408499 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:34.409333 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:34.410785 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:34.411358 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:34.411790 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:34.411804 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:34.411825 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 18:39:34.411872 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260620 18:39:34.412094 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.408260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.10.250.66:37457)
0620 18:39:34.408573 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:34.408575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:34.408581 (+ 6us) server_negotiation.cc:407] Connection header received
0620 18:39:34.408600 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:34.408601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:34.408610 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:34.408690 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:34.408906 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:34.409590 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:34.410880 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:34.411301 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:34.411394 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:34.411407 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:34.411957 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 18:39:34.412017 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":50,"threads_started":1}
I20260620 18:39:34.412304 18379 tablet_service.cc:1467] Processing CreateTablet for tablet 32ea187db3264657a306e1da8e244e48 (DEFAULT_TABLE table=test-table [id=f3c2e65f7d154ee1a8e0a665ddfa30ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:34.412452 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ea187db3264657a306e1da8e244e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:34.412570 18242 tablet_service.cc:1467] Processing CreateTablet for tablet 32ea187db3264657a306e1da8e244e48 (DEFAULT_TABLE table=test-table [id=f3c2e65f7d154ee1a8e0a665ddfa30ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:34.412688 18242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ea187db3264657a306e1da8e244e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:34.414081 18681 tablet_bootstrap.cc:492] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88: Bootstrap starting.
I20260620 18:39:34.414093 18682 tablet_bootstrap.cc:492] T 32ea187db3264657a306e1da8e244e48 P c4ef94b55fa94b9c90a32215dfb2fa2c: Bootstrap starting.
I20260620 18:39:34.414296 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.407970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:36941 (local address 127.0.0.1:44424)
0620 18:39:34.408096 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.408104 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:34.408142 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:34.408505 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:34.408505 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:34.408507 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:34.408539 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:34.408540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:34.409536 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:34.409537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:34.410732 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:34.410734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:34.411481 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:34.411483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:34.411550 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:34.411554 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:39:34.411615 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 18:39:34.411670 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":14,"thread_start_us":74,"threads_started":1}
I20260620 18:39:34.414624 18515 tablet_service.cc:1467] Processing CreateTablet for tablet 32ea187db3264657a306e1da8e244e48 (DEFAULT_TABLE table=test-table [id=f3c2e65f7d154ee1a8e0a665ddfa30ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:34.414691 18682 tablet_bootstrap.cc:654] T 32ea187db3264657a306e1da8e244e48 P c4ef94b55fa94b9c90a32215dfb2fa2c: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:34.414776 18515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ea187db3264657a306e1da8e244e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:34.414769 18681 tablet_bootstrap.cc:654] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:34.415521 18682 tablet_bootstrap.cc:492] T 32ea187db3264657a306e1da8e244e48 P c4ef94b55fa94b9c90a32215dfb2fa2c: No bootstrap required, opened a new log
I20260620 18:39:34.415585 18682 ts_tablet_manager.cc:1397] T 32ea187db3264657a306e1da8e244e48 P c4ef94b55fa94b9c90a32215dfb2fa2c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:39:34.415624 18681 tablet_bootstrap.cc:492] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88: No bootstrap required, opened a new log
I20260620 18:39:34.415683 18681 ts_tablet_manager.cc:1397] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:39:34.415731 18682 raft_consensus.cc:348] T 32ea187db3264657a306e1da8e244e48 P c4ef94b55fa94b9c90a32215dfb2fa2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:34.415859 18682 raft_consensus.cc:374] T 32ea187db3264657a306e1da8e244e48 P c4ef94b55fa94b9c90a32215dfb2fa2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:34.415882 18682 raft_consensus.cc:729] T 32ea187db3264657a306e1da8e244e48 P c4ef94b55fa94b9c90a32215dfb2fa2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ef94b55fa94b9c90a32215dfb2fa2c, State: Initialized, Role: FOLLOWER
I20260620 18:39:34.415841 18681 raft_consensus.cc:348] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:34.415918 18681 raft_consensus.cc:374] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:34.415939 18681 raft_consensus.cc:729] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4e832d399164153a9c69cae14100b88, State: Initialized, Role: FOLLOWER
I20260620 18:39:34.415933 18682 consensus_queue.cc:260] T 32ea187db3264657a306e1da8e244e48 P c4ef94b55fa94b9c90a32215dfb2fa2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:34.416003 18681 consensus_queue.cc:260] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:34.416152 18682 ts_tablet_manager.cc:1428] T 32ea187db3264657a306e1da8e244e48 P c4ef94b55fa94b9c90a32215dfb2fa2c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:34.416188 18681 ts_tablet_manager.cc:1428] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:34.416729 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.416599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.66:49751)
0620 18:39:34.416663 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.416706 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 18:39:34.417639 18445 heartbeater.cc:503] Master 127.10.250.124:35477 was elected leader, sending a full tablet report...
I20260620 18:39:34.417732 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.416970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.66:60967)
0620 18:39:34.417674 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.417703 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":687}
I20260620 18:39:34.417747 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.417625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.65:56677)
0620 18:39:34.417680 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.417716 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 18:39:34.417939 18687 tablet_bootstrap.cc:492] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: Bootstrap starting.
I20260620 18:39:34.418346 18687 tablet_bootstrap.cc:654] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:34.418646 18308 heartbeater.cc:503] Master 127.10.250.124:35477 was elected leader, sending a full tablet report...
I20260620 18:39:34.418910 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.418769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.65:48141)
0620 18:39:34.418853 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.418881 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:39:34.419247 18687 tablet_bootstrap.cc:492] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: No bootstrap required, opened a new log
I20260620 18:39:34.419293 18687 ts_tablet_manager.cc:1397] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:34.419387 18687 raft_consensus.cc:348] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:34.419441 18687 raft_consensus.cc:374] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:34.419454 18687 raft_consensus.cc:729] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3a1b3e7d4441c7ad394c3aa6409cc4, State: Initialized, Role: FOLLOWER
I20260620 18:39:34.419518 18687 consensus_queue.cc:260] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:34.419863 18687 ts_tablet_manager.cc:1428] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:34.419942 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.419833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.67:43675)
0620 18:39:34.419870 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.419906 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 18:39:34.420230 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.420119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.67:44949)
0620 18:39:34.420173 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.420202 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 18:39:34.420327 18581 heartbeater.cc:503] Master 127.10.250.124:35477 was elected leader, sending a full tablet report...
W20260620 18:39:34.455986 18311 tablet.cc:2369] T 32ea187db3264657a306e1da8e244e48 P c4ef94b55fa94b9c90a32215dfb2fa2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:39:34.484545 18584 tablet.cc:2369] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:39:34.511145 18686 raft_consensus.cc:482] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:34.511305 18686 raft_consensus.cc:504] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:34.511547 18686 leader_election.cc:288] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4ef94b55fa94b9c90a32215dfb2fa2c (127.10.250.65:33437), 5d3a1b3e7d4441c7ad394c3aa6409cc4 (127.10.250.67:36941)
I20260620 18:39:34.511946 18535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ea187db3264657a306e1da8e244e48" candidate_uuid: "e4e832d399164153a9c69cae14100b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" is_pre_election: true
I20260620 18:39:34.511973 18262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ea187db3264657a306e1da8e244e48" candidate_uuid: "e4e832d399164153a9c69cae14100b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" is_pre_election: true
I20260620 18:39:34.512049 18535 raft_consensus.cc:2454] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4e832d399164153a9c69cae14100b88 in term 0.
I20260620 18:39:34.512079 18262 raft_consensus.cc:2454] T 32ea187db3264657a306e1da8e244e48 P c4ef94b55fa94b9c90a32215dfb2fa2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4e832d399164153a9c69cae14100b88 in term 0.
I20260620 18:39:34.512257 18332 leader_election.cc:302] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4ef94b55fa94b9c90a32215dfb2fa2c, e4e832d399164153a9c69cae14100b88; no voters:
I20260620 18:39:34.512377 18686 raft_consensus.cc:2781] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:34.512421 18686 raft_consensus.cc:482] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:34.512446 18686 raft_consensus.cc:3037] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:34.513206 18686 raft_consensus.cc:504] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:34.513377 18686 leader_election.cc:288] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [CANDIDATE]: Term 1 election: Requested vote from peers c4ef94b55fa94b9c90a32215dfb2fa2c (127.10.250.65:33437), 5d3a1b3e7d4441c7ad394c3aa6409cc4 (127.10.250.67:36941)
I20260620 18:39:34.513569 18535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ea187db3264657a306e1da8e244e48" candidate_uuid: "e4e832d399164153a9c69cae14100b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4"
I20260620 18:39:34.513573 18262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ea187db3264657a306e1da8e244e48" candidate_uuid: "e4e832d399164153a9c69cae14100b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c"
I20260620 18:39:34.513645 18262 raft_consensus.cc:3037] T 32ea187db3264657a306e1da8e244e48 P c4ef94b55fa94b9c90a32215dfb2fa2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:34.513662 18535 raft_consensus.cc:3037] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:34.514258 18535 raft_consensus.cc:2454] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4e832d399164153a9c69cae14100b88 in term 1.
I20260620 18:39:34.514281 18262 raft_consensus.cc:2454] T 32ea187db3264657a306e1da8e244e48 P c4ef94b55fa94b9c90a32215dfb2fa2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4e832d399164153a9c69cae14100b88 in term 1.
I20260620 18:39:34.514489 18335 leader_election.cc:302] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3a1b3e7d4441c7ad394c3aa6409cc4, e4e832d399164153a9c69cae14100b88; no voters:
I20260620 18:39:34.514629 18686 raft_consensus.cc:2781] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:34.514810 18686 raft_consensus.cc:686] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [term 1 LEADER]: Becoming Leader. State: Replica: e4e832d399164153a9c69cae14100b88, State: Running, Role: LEADER
I20260620 18:39:34.514871 18686 consensus_queue.cc:237] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } }
I20260620 18:39:34.515399 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.515274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.66:56743)
0620 18:39:34.515330 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.515365 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 18:39:34.515587 18128 catalog_manager.cc:5495] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4e832d399164153a9c69cae14100b88 (127.10.250.66). New cstate: current_term: 1 leader_uuid: "e4e832d399164153a9c69cae14100b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4e832d399164153a9c69cae14100b88" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37457 } health_report { overall_health: HEALTHY } } }
I20260620 18:39:34.530033 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.528034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.10.250.66:37457)
0620 18:39:34.528096 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:34.528098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:34.528107 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:34.528148 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:34.528149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:34.528157 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:34.528195 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:34.528318 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:34.528784 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:34.529476 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:34.529658 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:34.529711 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:34.529819 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:34.529844 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:34.529880 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:34.529932 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 18:39:34.529951 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:39:34.531051 18262 raft_consensus.cc:1261] T 32ea187db3264657a306e1da8e244e48 P c4ef94b55fa94b9c90a32215dfb2fa2c [term 1 FOLLOWER]: Refusing update from remote peer e4e832d399164153a9c69cae14100b88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:34.531064 18535 raft_consensus.cc:1261] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 1 FOLLOWER]: Refusing update from remote peer e4e832d399164153a9c69cae14100b88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:34.531324 18690 consensus_queue.cc:1035] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4ef94b55fa94b9c90a32215dfb2fa2c" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 33437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:34.531426 18686 consensus_queue.cc:1035] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d3a1b3e7d4441c7ad394c3aa6409cc4" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:34.532191 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.531952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.65:57527)
0620 18:39:34.532114 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.532154 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260620 18:39:34.532621 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.532406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.67:44415)
0620 18:39:34.532553 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.532589 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260620 18:39:34.533425 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.533267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.66:39617)
0620 18:39:34.533363 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.533393 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 18:39:34.533651 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.533521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.67:55107)
0620 18:39:34.533596 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.533620 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 18:39:34.534041 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.533952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.65:57199)
0620 18:39:34.533989 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.534013 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 18:39:34.534418 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:34.534328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:46691 (local address 127.10.250.66:51627)
0620 18:39:34.534366 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.534389 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 18:39:34.534962 18695 mvcc.cc:205] Tried to move back new op lower bound from 7298993252477308928 to 7298993252413448192. Current Snapshot: MvccSnapshot[applied={T|T < 7298993252477308928 or (T in {7298993252477308928})}]
I20260620 18:39:34.535359 18129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:34.535444 18129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:34.536623 18129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 834ae29701f044b2a8e99747efffa486: Sending DeleteTablet for 3 replicas of tablet 32ea187db3264657a306e1da8e244e48
I20260620 18:39:34.537065 18379 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ea187db3264657a306e1da8e244e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:34 UTC) from {username='oryx', principal='oryx/127.10.250.124@KRBTEST.COM'} at 127.0.0.1:41524
I20260620 18:39:34.537112 18242 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ea187db3264657a306e1da8e244e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:34 UTC) from {username='oryx', principal='oryx/127.10.250.124@KRBTEST.COM'} at 127.0.0.1:37842
I20260620 18:39:34.537249 18703 tablet_replica.cc:330] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88: stopping tablet replica
I20260620 18:39:34.537266 18515 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ea187db3264657a306e1da8e244e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:34 UTC) from {username='oryx', principal='oryx/127.10.250.124@KRBTEST.COM'} at 127.0.0.1:44424
I20260620 18:39:34.537307 18703 raft_consensus.cc:2229] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:34.537376 18703 raft_consensus.cc:2258] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:34.537434 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 18091
I20260620 18:39:34.538226 18705 tablet_replica.cc:330] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: stopping tablet replica
I20260620 18:39:34.538329 18705 raft_consensus.cc:2229] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:34.538404 18705 raft_consensus.cc:2258] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:34.539085 18703 ts_tablet_manager.cc:1905] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:34.539160 18705 ts_tablet_manager.cc:1905] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:34.540637 18705 ts_tablet_manager.cc:1918] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:34.540712 18705 log.cc:1196] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/32ea187db3264657a306e1da8e244e48
I20260620 18:39:34.540949 18705 ts_tablet_manager.cc:1939] T 32ea187db3264657a306e1da8e244e48 P 5d3a1b3e7d4441c7ad394c3aa6409cc4: Deleting consensus metadata
I20260620 18:39:34.541186 18703 ts_tablet_manager.cc:1918] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:34.541244 18703 log.cc:1196] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/32ea187db3264657a306e1da8e244e48
I20260620 18:39:34.541251 18116 catalog_manager.cc:4841] TS 5d3a1b3e7d4441c7ad394c3aa6409cc4 (127.10.250.67:36941): tablet 32ea187db3264657a306e1da8e244e48 (table test-table [id=f3c2e65f7d154ee1a8e0a665ddfa30ba]) successfully deleted
I20260620 18:39:34.541473 18703 ts_tablet_manager.cc:1939] T 32ea187db3264657a306e1da8e244e48 P e4e832d399164153a9c69cae14100b88: Deleting consensus metadata
I20260620 18:39:34.541855 18115 catalog_manager.cc:4841] TS e4e832d399164153a9c69cae14100b88 (127.10.250.66:37457): tablet 32ea187db3264657a306e1da8e244e48 (table test-table [id=f3c2e65f7d154ee1a8e0a665ddfa30ba]) successfully deleted
W20260620 18:39:34.543515 18676 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:34.543364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:33437 (local address 127.0.0.1:37844)
0620 18:39:34.543428 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:34.543459 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:33437: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 18:39:34.543643 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 18316
W20260620 18:39:34.543680 18113 catalog_manager.cc:4584] TS c4ef94b55fa94b9c90a32215dfb2fa2c (127.10.250.65:33437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32ea187db3264657a306e1da8e244e48: Network error: Client connection negotiation failed: client connection to 127.10.250.65:33437: connect: Connection reset by peer (error 104)
I20260620 18:39:34.549674 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 18452
I20260620 18:39:34.554430 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 17809
I20260620 18:39:34.559051 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 17997
2026-06-20T18:39:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6368 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:34 dist-test-slave-wp6g krb5kdc[18708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:34 dist-test-slave-wp6g krb5kdc[18708](info): set up 2 sockets
Jun 20 18:39:34 dist-test-slave-wp6g krb5kdc[18708](info): commencing operation
krb5kdc: starting...
W20260620 18:39:35.607347 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:39:35 dist-test-slave-wp6g krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.126.keytab.
Entry for principal oryx/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.126.keytab.
I20260620 18:39:35.731081 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38757
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.126.keytab
--principal=oryx/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:38757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:35.818159 18721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:35.818377 18721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:35.818400 18721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:35.819551 18721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:35.819586 18721 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:39:35.819599 18721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:35.819613 18721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:35.819625 18721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:35.819638 18721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:35.821096 18721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:38757
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.126.keytab
--principal=oryx/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:35.821298 18721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:35.821476 18721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:35.826397 18727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:35.826507 18726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:35.826440 18729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:35.826682 18721 server_base.cc:1034] running on GCE node
I20260620 18:39:35.826957 18721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:35.827251 18721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:35.828481 18721 hybrid_clock.cc:648] HybridClock initialized: now 1781980775828459 us; error 37 us; skew 500 ppm
Jun 20 18:39:35 dist-test-slave-wp6g krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980775, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:35.831158 18721 init.cc:377] Logged in from keytab as oryx/127.10.250.126@KRBTEST.COM (short username oryx)
I20260620 18:39:35.831669 18721 webserver.cc:460] Webserver started at http://127.10.250.126:39979/ using document root <none> and password file <none>
I20260620 18:39:35.831934 18721 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:35.832010 18721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:35.832160 18721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:35.833292 18721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "6d888330b95741c992e8adad4ad3f733"
format_stamp: "Formatted at 2026-06-20 18:39:35 on dist-test-slave-wp6g"
server_key: "3bc7a20a0af872bf6ea444bf9fbf4ecf"
server_key_iv: "5f253e4369926139135f7f004ceaf496"
server_key_version: "encryptionkey@0"
I20260620 18:39:35.833693 18721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "6d888330b95741c992e8adad4ad3f733"
format_stamp: "Formatted at 2026-06-20 18:39:35 on dist-test-slave-wp6g"
server_key: "3bc7a20a0af872bf6ea444bf9fbf4ecf"
server_key_iv: "5f253e4369926139135f7f004ceaf496"
server_key_version: "encryptionkey@0"
I20260620 18:39:35.835244 18721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:35.836134 18736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:35.836382 18721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:35.836501 18721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "6d888330b95741c992e8adad4ad3f733"
format_stamp: "Formatted at 2026-06-20 18:39:35 on dist-test-slave-wp6g"
server_key: "3bc7a20a0af872bf6ea444bf9fbf4ecf"
server_key_iv: "5f253e4369926139135f7f004ceaf496"
server_key_version: "encryptionkey@0"
I20260620 18:39:35.836581 18721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:35.848469 18721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:35.851109 18721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:35.851303 18721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:35.856004 18721 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:38757
I20260620 18:39:35.856068 18798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:38757 every 8 connection(s)
I20260620 18:39:35.856410 18721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:35.857136 18799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:35.859750 18799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733: Bootstrap starting.
I20260620 18:39:35.860425 18799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:35.860749 18799 log.cc:824] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:35.861548 18799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733: No bootstrap required, opened a new log
I20260620 18:39:35.863765 18799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d888330b95741c992e8adad4ad3f733" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38757 } }
I20260620 18:39:35.863917 18799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:35.863940 18799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d888330b95741c992e8adad4ad3f733, State: Initialized, Role: FOLLOWER
I20260620 18:39:35.864102 18799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d888330b95741c992e8adad4ad3f733" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38757 } }
I20260620 18:39:35.864176 18799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:35.864219 18799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:35.864269 18799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:35.864845 18799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d888330b95741c992e8adad4ad3f733" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38757 } }
I20260620 18:39:35.864955 18799 leader_election.cc:302] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d888330b95741c992e8adad4ad3f733; no voters:
I20260620 18:39:35.865130 18799 leader_election.cc:288] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:35.865219 18802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:35.865394 18802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [term 1 LEADER]: Becoming Leader. State: Replica: 6d888330b95741c992e8adad4ad3f733, State: Running, Role: LEADER
I20260620 18:39:35.865415 18799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:35.865486 18802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d888330b95741c992e8adad4ad3f733" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38757 } }
I20260620 18:39:35.865887 18803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d888330b95741c992e8adad4ad3f733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d888330b95741c992e8adad4ad3f733" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38757 } } }
I20260620 18:39:35.865988 18803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:35.865873 18804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d888330b95741c992e8adad4ad3f733. Latest consensus state: current_term: 1 leader_uuid: "6d888330b95741c992e8adad4ad3f733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d888330b95741c992e8adad4ad3f733" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38757 } } }
I20260620 18:39:35.866257 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 18721
I20260620 18:39:35.866366 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:35.866596 11241 external_mini_cluster.cc:1428] Setting key 11ed882020d25895448e6e95b59564e5
I20260620 18:39:35.866305 18809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:35.867264 18809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:35.868086 18804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:35.869030 18809 catalog_manager.cc:1347] Generated new cluster ID: 62beb09b18d9492ba9a697689c2ad506
I20260620 18:39:35.869078 18809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:39:35 dist-test-slave-wp6g krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM
I20260620 18:39:35.875370 18809 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:35.876098 18809 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:35.877615 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:35.867185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.10.250.126:38757)
0620 18:39:35.867578 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:35.867588 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:35.867774 (+ 186us) server_negotiation.cc:407] Connection header received
0620 18:39:35.868059 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:35.868068 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:35.868225 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:35.868387 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:35.869159 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:35.870103 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:35.871013 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:35.871459 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:35.871580 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:35.873965 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:35.873970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:35.873970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:35.873984 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:35.875998 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:35.876425 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:35.876428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:35.876429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:35.876532 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:35.876747 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:35.876748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:35.876749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:35.876994 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:35.877193 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:35.877267 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 18:39:35.877430 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":72,"threads_started":1}
I20260620 18:39:35.881645 18809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733: Generated new TSK 0
I20260620 18:39:35.881860 18809 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.65.keytab.
Entry for principal oryx/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.65.keytab.
I20260620 18:39:35.910718 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38757
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.65.keytab
--principal=oryx/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:35.993602 18825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:35.993844 18825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:35.993884 18825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:35.995107 18825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:35.995178 18825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:35.995213 18825 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:39:35.995255 18825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:35.996765 18825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.65.keytab
--principal=oryx/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.250.126:38757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:35.997031 18825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:35.997262 18825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:36.002249 18830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:36.002360 18825 server_base.cc:1034] running on GCE node
W20260620 18:39:36.002449 18831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:36.002251 18833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:36.002728 18825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:36.002938 18825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:36.004176 18825 hybrid_clock.cc:648] HybridClock initialized: now 1781980776004160 us; error 26 us; skew 500 ppm
Jun 20 18:39:36 dist-test-slave-wp6g krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980776, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:36.007093 18825 init.cc:377] Logged in from keytab as oryx/127.10.250.65@KRBTEST.COM (short username oryx)
I20260620 18:39:36.007648 18825 webserver.cc:460] Webserver started at http://127.10.250.65:44251/ using document root <none> and password file <none>
I20260620 18:39:36.007885 18825 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:36.007968 18825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:36.008128 18825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:36.009222 18825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "0ea5e748bc214b2a9f5829ab5f744ebb"
format_stamp: "Formatted at 2026-06-20 18:39:36 on dist-test-slave-wp6g"
server_key: "ff327e7ebf83a2c6e176c83ee5bd8f83"
server_key_iv: "6236ece79776a7f3981d9cd0c6e2b29c"
server_key_version: "encryptionkey@0"
I20260620 18:39:36.009588 18825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "0ea5e748bc214b2a9f5829ab5f744ebb"
format_stamp: "Formatted at 2026-06-20 18:39:36 on dist-test-slave-wp6g"
server_key: "ff327e7ebf83a2c6e176c83ee5bd8f83"
server_key_iv: "6236ece79776a7f3981d9cd0c6e2b29c"
server_key_version: "encryptionkey@0"
I20260620 18:39:36.011031 18825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:39:36.011761 18840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:36.011992 18825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:36.012120 18825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "0ea5e748bc214b2a9f5829ab5f744ebb"
format_stamp: "Formatted at 2026-06-20 18:39:36 on dist-test-slave-wp6g"
server_key: "ff327e7ebf83a2c6e176c83ee5bd8f83"
server_key_iv: "6236ece79776a7f3981d9cd0c6e2b29c"
server_key_version: "encryptionkey@0"
I20260620 18:39:36.012207 18825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:36.035790 18825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:36.038147 18825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:36.038350 18825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:36.038990 18825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:36.039054 18825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:36.039098 18825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:36.039130 18825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:39:36 dist-test-slave-wp6g krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980776, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM
I20260620 18:39:36.046841 18825 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:32873
I20260620 18:39:36.046932 18955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:32873 every 8 connection(s)
I20260620 18:39:36.047291 18825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:36.049213 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.039620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:54183 (local address 127.10.250.126:38757)
0620 18:39:36.039685 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.039688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.039883 (+ 195us) server_negotiation.cc:407] Connection header received
0620 18:39:36.040484 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.040486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.040500 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.040569 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:36.041273 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.042142 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.043333 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.043720 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.043818 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.046162 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:36.046163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:36.046164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:36.046196 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:36.048022 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.048447 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.048447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.048448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.048497 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.048674 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.048674 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.048674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.048777 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:36.048828 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.049071 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.049148 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 18:39:36.049414 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.039582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38757 (local address 127.10.250.65:54183)
0620 18:39:36.039742 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.039786 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.040364 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.040650 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.040653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.040969 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:36.041168 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.041174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.042246 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.042247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.043258 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.043261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.043844 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.043845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.043995 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.044586 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:36.044595 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:36.046041 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:36.048100 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:36.048104 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:36.048106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:36.048377 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:36.048538 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:36.048538 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:36.048539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:36.048625 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:36.048837 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:36.048842 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:36.048958 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.049041 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.049162 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260620 18:39:36.049954 18956 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38757
I20260620 18:39:36.050031 18956 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:36.050215 18956 heartbeater.cc:511] Master 127.10.250.126:38757 requested a full tablet report, sending...
I20260620 18:39:36.050806 18752 ts_manager.cc:194] Registered new tserver with Master: 0ea5e748bc214b2a9f5829ab5f744ebb (127.10.250.65:32873)
I20260620 18:39:36.051676 18752 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:54183
I20260620 18:39:36.056453 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 18825
I20260620 18:39:36.056568 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:36.056720 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.051017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:41373 (local address 127.10.250.126:38757)
0620 18:39:36.051057 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.051060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.051172 (+ 112us) server_negotiation.cc:407] Connection header received
0620 18:39:36.051200 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.051201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.051211 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.051272 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:36.051561 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.052216 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.053008 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.053190 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.053284 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.054115 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:36.054116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:36.054117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:36.054123 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:36.055573 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.055887 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.055888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.055888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.055947 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.056169 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.056170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.056170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.056330 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:36.056405 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.056545 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.056627 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:39:36.056783 11241 external_mini_cluster.cc:1428] Setting key d518545495a988eccb5ce214cf97a5a9
I20260620 18:39:36.056833 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.050971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38757 (local address 127.10.250.65:41373)
0620 18:39:36.051093 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.051103 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.051165 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.051286 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.051286 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.051439 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:36.051500 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.051501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.052322 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.052323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.052925 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.052927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.053258 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.053259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.053304 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.053722 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:36.053726 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:36.054038 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:36.055642 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:36.055643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:36.055643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:36.055801 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:36.056036 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:36.056037 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:36.056037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:36.056102 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:36.056399 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:36.056399 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:36.056446 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.056500 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.056652 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for oryx/127.10.250.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.66.keytab.
Entry for principal oryx/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.66.keytab.
I20260620 18:39:36.082684 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38757
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.66.keytab
--principal=oryx/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:36.165278 18965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:36.165505 18965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:36.165547 18965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:36.166826 18965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:36.166890 18965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:36.166924 18965 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:39:36.166963 18965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:36.168490 18965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.66.keytab
--principal=oryx/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.250.126:38757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:36.168768 18965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:36.169005 18965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:36.174248 18971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:36.174255 18970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:36.174248 18973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:36.174561 18965 server_base.cc:1034] running on GCE node
I20260620 18:39:36.174822 18965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:36.175086 18965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:36.176303 18965 hybrid_clock.cc:648] HybridClock initialized: now 1781980776176298 us; error 38 us; skew 500 ppm
Jun 20 18:39:36 dist-test-slave-wp6g krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980776, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:36.179236 18965 init.cc:377] Logged in from keytab as oryx/127.10.250.66@KRBTEST.COM (short username oryx)
I20260620 18:39:36.179735 18965 webserver.cc:460] Webserver started at http://127.10.250.66:44743/ using document root <none> and password file <none>
I20260620 18:39:36.180015 18965 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:36.180089 18965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:36.180240 18965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:36.181353 18965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed"
format_stamp: "Formatted at 2026-06-20 18:39:36 on dist-test-slave-wp6g"
server_key: "df632374c455c7703799c357409c6a34"
server_key_iv: "12e13ed3f27d49506373f49e9635dd1b"
server_key_version: "encryptionkey@0"
I20260620 18:39:36.181721 18965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed"
format_stamp: "Formatted at 2026-06-20 18:39:36 on dist-test-slave-wp6g"
server_key: "df632374c455c7703799c357409c6a34"
server_key_iv: "12e13ed3f27d49506373f49e9635dd1b"
server_key_version: "encryptionkey@0"
I20260620 18:39:36.183177 18965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:36.183933 18980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:36.184120 18965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:36.184226 18965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed"
format_stamp: "Formatted at 2026-06-20 18:39:36 on dist-test-slave-wp6g"
server_key: "df632374c455c7703799c357409c6a34"
server_key_iv: "12e13ed3f27d49506373f49e9635dd1b"
server_key_version: "encryptionkey@0"
I20260620 18:39:36.184304 18965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:36.205410 18965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:36.208003 18965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:36.208184 18965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:36.208707 18965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:36.208773 18965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:36.208817 18965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:36.208847 18965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:39:36 dist-test-slave-wp6g krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980776, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM
I20260620 18:39:36.216161 18965 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:40357
I20260620 18:39:36.216542 18965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:36.216246 19095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:40357 every 8 connection(s)
I20260620 18:39:36.218037 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 18965
I20260620 18:39:36.218171 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:36.218391 11241 external_mini_cluster.cc:1428] Setting key f549095eee7fed5a1db3e97d6ab6401e
I20260620 18:39:36.219583 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.209306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:52687 (local address 127.10.250.126:38757)
0620 18:39:36.209428 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.209431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.209604 (+ 173us) server_negotiation.cc:407] Connection header received
0620 18:39:36.210117 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.210119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.210133 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.210219 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:36.211183 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.212060 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.213432 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.213687 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.213769 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.216571 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:36.216572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:36.216572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:36.216582 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:36.217981 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.218460 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.218461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.218461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.218574 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.218819 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.218820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.218820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.218967 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:36.219032 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.219347 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.219443 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:39:36.219611 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.209207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38757 (local address 127.10.250.66:52687)
0620 18:39:36.209490 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.209520 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.209979 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.210319 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.210323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.210765 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:36.211072 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.211078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.212176 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.212177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.213323 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.213326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.213803 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.213805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.213972 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.214563 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:36.214573 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:36.216468 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:36.218089 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:36.218096 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:36.218098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:36.218367 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:36.218626 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:36.218627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:36.218627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:36.218756 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:36.219038 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:36.219044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:36.219203 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.219304 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.219437 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":79,"threads_started":1}
I20260620 18:39:36.220988 19096 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38757
I20260620 18:39:36.221078 19096 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:36.221256 19096 heartbeater.cc:511] Master 127.10.250.126:38757 requested a full tablet report, sending...
I20260620 18:39:36.221684 18752 ts_manager.cc:194] Registered new tserver with Master: 9bf4bbd6fdb54e81aedd96b25f7a6fed (127.10.250.66:40357)
I20260620 18:39:36.222064 18752 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:52687
WARNING: no policy specified for oryx/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.250.67@KRBTEST.COM" created.
I20260620 18:39:36.232908 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.221168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:57663 (local address 127.10.250.126:38757)
0620 18:39:36.221211 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.221213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.226366 (+ 5153us) server_negotiation.cc:407] Connection header received
0620 18:39:36.226392 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.226393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.226402 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.226445 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:36.226854 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.227418 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.228217 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.228431 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.228496 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.229826 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:36.229827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:36.229827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:36.229834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:36.231614 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.231983 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.231984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.231985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.232047 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.232266 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.232268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.232268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.232444 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:36.232552 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.232725 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.232809 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:39:36.232947 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.221164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38757 (local address 127.10.250.66:57663)
0620 18:39:36.226249 (+ 5085us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.226261 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.226328 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.226465 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.226466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.226682 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:36.226777 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.226779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.227508 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.227509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.228119 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.228121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.228524 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.228525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.228621 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.229239 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:36.229246 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:36.229704 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:36.231710 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:36.231711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:36.231711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:36.231905 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:36.232116 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:36.232117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:36.232117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:36.232173 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:36.232544 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:36.232545 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:36.232618 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.232697 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.232809 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5053,"thread_start_us":61,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.67.keytab.
Entry for principal oryx/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.67.keytab.
I20260620 18:39:36.245785 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38757
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.67.keytab
--principal=oryx/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:36.328662 19105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:36.328888 19105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:36.328922 19105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:36.330127 19105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:36.330171 19105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:36.330193 19105 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:39:36.330211 19105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:36.331787 19105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.67.keytab
--principal=oryx/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.250.126:38757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:36.332049 19105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:36.332301 19105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:36.337467 19113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:36.337467 19110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:36.337527 19111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:36.337590 19105 server_base.cc:1034] running on GCE node
I20260620 18:39:36.337888 19105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:36.338186 19105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:36.339349 19105 hybrid_clock.cc:648] HybridClock initialized: now 1781980776339335 us; error 33 us; skew 500 ppm
Jun 20 18:39:36 dist-test-slave-wp6g krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980776, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:36.342010 19105 init.cc:377] Logged in from keytab as oryx/127.10.250.67@KRBTEST.COM (short username oryx)
I20260620 18:39:36.342464 19105 webserver.cc:460] Webserver started at http://127.10.250.67:33435/ using document root <none> and password file <none>
I20260620 18:39:36.342707 19105 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:36.342752 19105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:36.342873 19105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:36.343865 19105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "6a1beed494f744b6a28515dd3e65a324"
format_stamp: "Formatted at 2026-06-20 18:39:36 on dist-test-slave-wp6g"
server_key: "1c0264e0c1ae78529753ca4d48cd0a7d"
server_key_iv: "df20b908ec010779a0796acf840df7b9"
server_key_version: "encryptionkey@0"
I20260620 18:39:36.344165 19105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "6a1beed494f744b6a28515dd3e65a324"
format_stamp: "Formatted at 2026-06-20 18:39:36 on dist-test-slave-wp6g"
server_key: "1c0264e0c1ae78529753ca4d48cd0a7d"
server_key_iv: "df20b908ec010779a0796acf840df7b9"
server_key_version: "encryptionkey@0"
I20260620 18:39:36.345476 19105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:36.346251 19120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:36.346453 19105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:36.346596 19105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "6a1beed494f744b6a28515dd3e65a324"
format_stamp: "Formatted at 2026-06-20 18:39:36 on dist-test-slave-wp6g"
server_key: "1c0264e0c1ae78529753ca4d48cd0a7d"
server_key_iv: "df20b908ec010779a0796acf840df7b9"
server_key_version: "encryptionkey@0"
I20260620 18:39:36.346661 19105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:36.353888 19105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:36.356304 19105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:36.356479 19105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:36.357025 19105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:36.357080 19105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:36.357108 19105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:36.357122 19105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:36.364573 19105 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:33945
I20260620 18:39:36.364624 19235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:33945 every 8 connection(s)
Jun 20 18:39:36 dist-test-slave-wp6g krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980776, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM
I20260620 18:39:36.364946 19105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:36.368611 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.357643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:44639 (local address 127.10.250.126:38757)
0620 18:39:36.357723 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.357726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.357882 (+ 156us) server_negotiation.cc:407] Connection header received
0620 18:39:36.358407 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.358408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.358422 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.358549 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:36.359505 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.360370 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.362126 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.362472 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.362643 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.365476 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:36.365478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:36.365479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:36.365487 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:36.367171 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.367625 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.367626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.367627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.367701 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.367963 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.367965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.367965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.368088 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:36.368156 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.368467 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.368543 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:39:36.368781 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.357528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38757 (local address 127.10.250.67:44639)
0620 18:39:36.357734 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.357777 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.358303 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.358642 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.358645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.359033 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:36.359390 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.359396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.360497 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.360498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.362027 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.362031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.362670 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.362672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.362839 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.363484 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:36.363498 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:36.365346 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:36.367239 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:36.367245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:36.367246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:36.367520 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:36.367776 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:36.367776 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:36.367777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:36.367879 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:36.368185 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:36.368191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:36.368320 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.368434 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.368568 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":101,"threads_started":1}
I20260620 18:39:36.369289 19236 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38757
I20260620 18:39:36.369376 19236 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:36.369581 19236 heartbeater.cc:511] Master 127.10.250.126:38757 requested a full tablet report, sending...
I20260620 18:39:36.370061 18753 ts_manager.cc:194] Registered new tserver with Master: 6a1beed494f744b6a28515dd3e65a324 (127.10.250.67:33945)
I20260620 18:39:36.370610 18753 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:44639
I20260620 18:39:36.370621 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 19105
I20260620 18:39:36.370720 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:36.370926 11241 external_mini_cluster.cc:1428] Setting key 36284ecaeb845278bd79e06762e72057
I20260620 18:39:36.372355 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:39:36.376729 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.370399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:51837 (local address 127.10.250.126:38757)
0620 18:39:36.370445 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.370448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.371162 (+ 714us) server_negotiation.cc:407] Connection header received
0620 18:39:36.371261 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.371262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.371272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.371328 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:36.371669 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.372215 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.372907 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.373110 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.373190 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.374155 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:36.374156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:36.374157 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:36.374163 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:36.375611 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.375952 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.375953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.375954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.376014 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.376206 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.376207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.376208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.376322 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:36.376398 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.376570 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.376637 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:39:36.376744 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.370307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38757 (local address 127.10.250.67:51837)
0620 18:39:36.371096 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.371108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.371195 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.371334 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.371334 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.371517 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:36.371597 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.371598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.372303 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.372304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.372816 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.372818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.373192 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.373193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.373267 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.373726 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:36.373736 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:36.374048 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:36.375707 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:36.375707 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:36.375708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:36.375877 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:36.376080 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:36.376081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:36.376081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:36.376143 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:36.376387 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:36.376388 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:36.376452 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.376532 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.376638 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":17,"thread_start_us":83,"threads_started":1}
Jun 20 18:39:36 dist-test-slave-wp6g krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980776, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:39:36 dist-test-slave-wp6g krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:36 dist-test-slave-wp6g krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:39:36.395565 18819 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:36.389702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33810 (local address 127.10.250.126:38757)
0620 18:39:36.389768 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.389770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.389780 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:39:36.389850 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.389851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.389860 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.389920 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:36.390590 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.391082 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.391849 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.392047 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.392100 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.395458 (+ 3358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33810: BlockingRecv error: recv got EOF from 127.0.0.1:33810 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 18:39:36 dist-test-slave-wp6g krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980776, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM
I20260620 18:39:36.402827 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.396243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.10.250.126:38757)
0620 18:39:36.396298 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.396300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.396413 (+ 113us) server_negotiation.cc:407] Connection header received
0620 18:39:36.396450 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.396450 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.396458 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.396512 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:36.396942 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.397395 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.398037 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.398203 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.398250 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.400354 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:36.400355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:36.400356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:36.400364 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:36.401632 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.402008 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.402009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.402010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.402072 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.402274 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.402275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.402275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.402397 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:36.402511 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.402637 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.402714 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:39:36.404295 18753 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:36.405877 18753 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:36.412569 19258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.408325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:33945 (local address 127.0.0.1:43886)
0620 18:39:36.408649 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.408659 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.408725 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.408989 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.408992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.408994 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:36.409071 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.409073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.410329 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.410330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.411422 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.411425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.412209 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.412210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.412308 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.412324 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.412407 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.412466 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":100,"threads_started":1}
I20260620 18:39:36.412734 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.408525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:40357 (local address 127.0.0.1:51934)
0620 18:39:36.408747 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.408757 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.408801 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.409272 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.409273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.409276 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:36.409309 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.409310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.410550 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.410551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.411519 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.411522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.412422 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.412423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.412474 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.412477 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.412532 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.412562 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260620 18:39:36.412775 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.408417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.10.250.67:33945)
0620 18:39:36.408604 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.408609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.408725 (+ 116us) server_negotiation.cc:407] Connection header received
0620 18:39:36.408802 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.408805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.408823 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.408899 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:36.409155 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.410242 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.411514 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.412102 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.412206 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.412292 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.412620 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.412674 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":71,"threads_started":1}
I20260620 18:39:36.413170 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.408321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:32873 (local address 127.0.0.1:37198)
0620 18:39:36.408485 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.408528 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.408594 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.409048 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.409049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.409051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:36.409109 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.409111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.410023 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.410024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.411465 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.411467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.412913 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.412914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.412981 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.412985 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.413067 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.413108 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":1,"thread_start_us":59,"threads_started":1}
I20260620 18:39:36.413233 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.408515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.10.250.65:32873)
0620 18:39:36.408865 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.408870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.408880 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:39:36.408928 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.408930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.408951 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.409038 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:36.409168 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.409934 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.411555 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.412341 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.412423 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.412494 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.413087 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.413129 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":84,"threads_started":1}
I20260620 18:39:36.413523 19170 tablet_service.cc:1467] Processing CreateTablet for tablet d01b57881bea4199be71beb4815eb1ae (DEFAULT_TABLE table=test-table [id=b19bded3347d49cb969115439dd0e0b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:36.413923 19170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01b57881bea4199be71beb4815eb1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:36.413894 18890 tablet_service.cc:1467] Processing CreateTablet for tablet d01b57881bea4199be71beb4815eb1ae (DEFAULT_TABLE table=test-table [id=b19bded3347d49cb969115439dd0e0b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:36.414243 18890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01b57881bea4199be71beb4815eb1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:36.412822 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.408745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51934 (local address 127.10.250.66:40357)
0620 18:39:36.408937 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.408942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.408951 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:36.408988 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.408990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.409004 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.409379 (+ 375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:36.409440 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.410452 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.411623 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.412333 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.412415 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.412494 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.412548 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.412592 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":84,"threads_started":1}
I20260620 18:39:36.415184 19030 tablet_service.cc:1467] Processing CreateTablet for tablet d01b57881bea4199be71beb4815eb1ae (DEFAULT_TABLE table=test-table [id=b19bded3347d49cb969115439dd0e0b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:36.415416 19030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01b57881bea4199be71beb4815eb1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:36.417866 19264 tablet_bootstrap.cc:492] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324: Bootstrap starting.
I20260620 18:39:36.418358 19265 tablet_bootstrap.cc:492] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb: Bootstrap starting.
I20260620 18:39:36.418551 19264 tablet_bootstrap.cc:654] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:36.418885 19266 tablet_bootstrap.cc:492] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Bootstrap starting.
I20260620 18:39:36.418908 19264 log.cc:824] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:36.419085 19265 tablet_bootstrap.cc:654] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:36.419420 19265 log.cc:824] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:36.419644 19266 tablet_bootstrap.cc:654] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:36.419745 19264 tablet_bootstrap.cc:492] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324: No bootstrap required, opened a new log
I20260620 18:39:36.419821 19264 ts_tablet_manager.cc:1397] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:39:36.419973 19266 log.cc:824] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:36.420318 19265 tablet_bootstrap.cc:492] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb: No bootstrap required, opened a new log
I20260620 18:39:36.420389 19265 ts_tablet_manager.cc:1397] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:39:36.420719 19266 tablet_bootstrap.cc:492] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed: No bootstrap required, opened a new log
I20260620 18:39:36.420796 19266 ts_tablet_manager.cc:1397] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:39:36.421927 19264 raft_consensus.cc:348] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } }
I20260620 18:39:36.422109 19264 raft_consensus.cc:374] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:36.422143 19264 raft_consensus.cc:729] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1beed494f744b6a28515dd3e65a324, State: Initialized, Role: FOLLOWER
I20260620 18:39:36.422266 19264 consensus_queue.cc:260] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } }
I20260620 18:39:36.422241 19266 raft_consensus.cc:348] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } }
I20260620 18:39:36.422417 19266 raft_consensus.cc:374] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:36.422461 19266 raft_consensus.cc:729] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf4bbd6fdb54e81aedd96b25f7a6fed, State: Initialized, Role: FOLLOWER
I20260620 18:39:36.422353 19265 raft_consensus.cc:348] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } }
I20260620 18:39:36.422505 19265 raft_consensus.cc:374] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:36.422611 19264 ts_tablet_manager.cc:1428] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 18:39:36.422626 19265 raft_consensus.cc:729] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea5e748bc214b2a9f5829ab5f744ebb, State: Initialized, Role: FOLLOWER
I20260620 18:39:36.422602 19266 consensus_queue.cc:260] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } }
I20260620 18:39:36.422729 19236 heartbeater.cc:503] Master 127.10.250.126:38757 was elected leader, sending a full tablet report...
I20260620 18:39:36.422869 19266 ts_tablet_manager.cc:1428] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:39:36.423000 19096 heartbeater.cc:503] Master 127.10.250.126:38757 was elected leader, sending a full tablet report...
I20260620 18:39:36.423278 19265 consensus_queue.cc:260] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } }
I20260620 18:39:36.423561 19265 ts_tablet_manager.cc:1428] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:36.423638 18956 heartbeater.cc:503] Master 127.10.250.126:38757 was elected leader, sending a full tablet report...
I20260620 18:39:36.449620 19272 raft_consensus.cc:482] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:36.449743 19272 raft_consensus.cc:504] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } }
I20260620 18:39:36.450107 19272 leader_election.cc:288] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a1beed494f744b6a28515dd3e65a324 (127.10.250.67:33945), 9bf4bbd6fdb54e81aedd96b25f7a6fed (127.10.250.66:40357)
I20260620 18:39:36.452970 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.450357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:36077 (local address 127.10.250.67:33945)
0620 18:39:36.450406 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.450409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.450418 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:36.450464 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.450465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.450503 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.450573 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:36.450767 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.451246 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.452216 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.452577 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.452666 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.452675 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.452857 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.452893 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 18:39:36.452982 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.450274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:33945 (local address 127.10.250.65:36077)
0620 18:39:36.450347 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.450359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.450408 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.450598 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.450600 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.450601 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:36.450678 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.450680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.451344 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.451345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.452107 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.452121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.452665 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.452666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.452771 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.452774 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.452849 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.452892 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 18:39:36.453076 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.450354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:35425 (local address 127.10.250.66:40357)
0620 18:39:36.450399 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.450402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.450591 (+ 189us) server_negotiation.cc:407] Connection header received
0620 18:39:36.450718 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.450719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.450729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.450804 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:36.450973 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.451435 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.452328 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.452701 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.452784 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.452792 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.452958 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.452994 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:39:36.453305 19273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.450284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:40357 (local address 127.10.250.65:35425)
0620 18:39:36.450503 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.450534 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.450634 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.450807 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.450808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.450809 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:36.450883 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.450885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.451518 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.451519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.452233 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.452235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.452784 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.452785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.452855 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.452858 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.452946 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.452988 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":1,"thread_start_us":149,"threads_started":1}
I20260620 18:39:36.453339 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01b57881bea4199be71beb4815eb1ae" candidate_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1beed494f744b6a28515dd3e65a324" is_pre_election: true
I20260620 18:39:36.453534 19190 raft_consensus.cc:2454] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea5e748bc214b2a9f5829ab5f744ebb in term 0.
I20260620 18:39:36.453593 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01b57881bea4199be71beb4815eb1ae" candidate_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" is_pre_election: true
I20260620 18:39:36.453730 19050 raft_consensus.cc:2454] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea5e748bc214b2a9f5829ab5f744ebb in term 0.
I20260620 18:39:36.453824 18842 leader_election.cc:302] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea5e748bc214b2a9f5829ab5f744ebb, 6a1beed494f744b6a28515dd3e65a324; no voters:
I20260620 18:39:36.454047 19272 raft_consensus.cc:2781] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:36.454094 19272 raft_consensus.cc:482] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:36.454111 19272 raft_consensus.cc:3037] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:36.455085 19272 raft_consensus.cc:504] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } }
I20260620 18:39:36.455235 19272 leader_election.cc:288] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [CANDIDATE]: Term 1 election: Requested vote from peers 6a1beed494f744b6a28515dd3e65a324 (127.10.250.67:33945), 9bf4bbd6fdb54e81aedd96b25f7a6fed (127.10.250.66:40357)
I20260620 18:39:36.455380 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01b57881bea4199be71beb4815eb1ae" candidate_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1beed494f744b6a28515dd3e65a324"
I20260620 18:39:36.455468 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01b57881bea4199be71beb4815eb1ae" candidate_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed"
I20260620 18:39:36.455497 19190 raft_consensus.cc:3037] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:36.455534 19050 raft_consensus.cc:3037] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:36.456372 19050 raft_consensus.cc:2454] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea5e748bc214b2a9f5829ab5f744ebb in term 1.
I20260620 18:39:36.456372 19190 raft_consensus.cc:2454] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea5e748bc214b2a9f5829ab5f744ebb in term 1.
I20260620 18:39:36.456580 18842 leader_election.cc:302] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea5e748bc214b2a9f5829ab5f744ebb, 6a1beed494f744b6a28515dd3e65a324; no voters:
I20260620 18:39:36.456687 19272 raft_consensus.cc:2781] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:36.456851 19272 raft_consensus.cc:686] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 LEADER]: Becoming Leader. State: Replica: 0ea5e748bc214b2a9f5829ab5f744ebb, State: Running, Role: LEADER
I20260620 18:39:36.456935 19272 consensus_queue.cc:237] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } }
I20260620 18:39:36.457896 18752 catalog_manager.cc:5495] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ea5e748bc214b2a9f5829ab5f744ebb (127.10.250.65). New cstate: current_term: 1 leader_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } health_report { overall_health: UNKNOWN } } }
W20260620 18:39:36.467151 19097 tablet.cc:2369] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:39:36.470626 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.468178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.10.250.65:32873)
0620 18:39:36.468225 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.468227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.468235 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:39:36.468297 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.468298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.468316 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.468382 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:36.468518 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.469109 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.469838 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.470063 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.470137 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.470251 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:36.470345 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:36.470413 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.470465 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.470482 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 18:39:36.472436 19050 raft_consensus.cc:1261] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Refusing update from remote peer 0ea5e748bc214b2a9f5829ab5f744ebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:36.472471 19190 raft_consensus.cc:1261] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324 [term 1 FOLLOWER]: Refusing update from remote peer 0ea5e748bc214b2a9f5829ab5f744ebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:36.472766 19272 consensus_queue.cc:1035] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:36.472918 19274 consensus_queue.cc:1035] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:36.475520 19279 mvcc.cc:205] Tried to move back new op lower bound from 7298993260428386304 to 7298993260368310272. Current Snapshot: MvccSnapshot[applied={T|T < 7298993260428386304}]
I20260620 18:39:36.475684 19278 mvcc.cc:205] Tried to move back new op lower bound from 7298993260428386304 to 7298993260368310272. Current Snapshot: MvccSnapshot[applied={T|T < 7298993260428386304}]
I20260620 18:39:36.477766 18752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:36.477870 18752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:36.478752 18752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733: Sending DeleteTablet for 3 replicas of tablet d01b57881bea4199be71beb4815eb1ae
I20260620 18:39:36.479064 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet d01b57881bea4199be71beb4815eb1ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:36 UTC) from {username='oryx', principal='oryx/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:37198
I20260620 18:39:36.479089 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet d01b57881bea4199be71beb4815eb1ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:36 UTC) from {username='oryx', principal='oryx/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:43886
I20260620 18:39:36.479493 19030 tablet_service.cc:1514] Processing DeleteTablet for tablet d01b57881bea4199be71beb4815eb1ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:36 UTC) from {username='oryx', principal='oryx/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:51934
I20260620 18:39:36.479542 19287 tablet_replica.cc:330] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324: stopping tablet replica
I20260620 18:39:36.479640 19287 raft_consensus.cc:2229] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:36.479759 19287 raft_consensus.cc:2258] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:36.479717 18752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:36.479869 18752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:36.479764 19289 tablet_replica.cc:330] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed: stopping tablet replica
I20260620 18:39:36.480103 19289 raft_consensus.cc:2229] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:36.480263 19289 raft_consensus.cc:2258] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:36.480754 19289 ts_tablet_manager.cc:1905] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:36.481144 19170 tablet_service.cc:1467] Processing CreateTablet for tablet 938e9fc88869444887e42458184b3c81 (DEFAULT_TABLE table=test-table [id=61686d053de240a9968a0e78493b07d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:36.481143 18890 tablet_service.cc:1467] Processing CreateTablet for tablet 938e9fc88869444887e42458184b3c81 (DEFAULT_TABLE table=test-table [id=61686d053de240a9968a0e78493b07d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:36.481266 18890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938e9fc88869444887e42458184b3c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:36.481274 19170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938e9fc88869444887e42458184b3c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:36.482419 19264 tablet_bootstrap.cc:492] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324: Bootstrap starting.
I20260620 18:39:36.482640 19265 tablet_bootstrap.cc:492] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb: Bootstrap starting.
I20260620 18:39:36.482897 19264 tablet_bootstrap.cc:654] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:36.482909 19289 ts_tablet_manager.cc:1918] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:36.482977 19289 log.cc:1196] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/d01b57881bea4199be71beb4815eb1ae
I20260620 18:39:36.483109 19265 tablet_bootstrap.cc:654] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:36.483234 19289 ts_tablet_manager.cc:1939] T d01b57881bea4199be71beb4815eb1ae P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Deleting consensus metadata
I20260620 18:39:36.483618 19030 tablet_service.cc:1467] Processing CreateTablet for tablet 938e9fc88869444887e42458184b3c81 (DEFAULT_TABLE table=test-table [id=61686d053de240a9968a0e78493b07d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:36.483642 18739 catalog_manager.cc:4841] TS 9bf4bbd6fdb54e81aedd96b25f7a6fed (127.10.250.66:40357): tablet d01b57881bea4199be71beb4815eb1ae (table test-table [id=b19bded3347d49cb969115439dd0e0b5]) successfully deleted
I20260620 18:39:36.483760 19030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938e9fc88869444887e42458184b3c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:36.484601 19266 tablet_bootstrap.cc:492] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Bootstrap starting.
I20260620 18:39:36.484764 19265 tablet_bootstrap.cc:492] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb: No bootstrap required, opened a new log
I20260620 18:39:36.484815 19265 ts_tablet_manager.cc:1397] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:39:36.484987 19266 tablet_bootstrap.cc:654] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:36.484969 19265 raft_consensus.cc:348] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.485049 19265 raft_consensus.cc:374] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:36.485078 19265 raft_consensus.cc:729] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea5e748bc214b2a9f5829ab5f744ebb, State: Initialized, Role: FOLLOWER
I20260620 18:39:36.485121 19265 consensus_queue.cc:260] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.485401 19265 ts_tablet_manager.cc:1428] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:36.485852 19287 ts_tablet_manager.cc:1905] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:36.485908 19266 tablet_bootstrap.cc:492] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: No bootstrap required, opened a new log
I20260620 18:39:36.485949 19266 ts_tablet_manager.cc:1397] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:36.486107 19266 raft_consensus.cc:348] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.486183 19266 raft_consensus.cc:374] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:36.486204 19266 raft_consensus.cc:729] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf4bbd6fdb54e81aedd96b25f7a6fed, State: Initialized, Role: FOLLOWER
I20260620 18:39:36.486246 19266 consensus_queue.cc:260] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.486340 19266 ts_tablet_manager.cc:1428] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:36.487536 19287 ts_tablet_manager.cc:1918] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:36.487620 19287 log.cc:1196] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/d01b57881bea4199be71beb4815eb1ae
I20260620 18:39:36.487920 19287 ts_tablet_manager.cc:1939] T d01b57881bea4199be71beb4815eb1ae P 6a1beed494f744b6a28515dd3e65a324: Deleting consensus metadata
I20260620 18:39:36.488348 18738 catalog_manager.cc:4841] TS 6a1beed494f744b6a28515dd3e65a324 (127.10.250.67:33945): tablet d01b57881bea4199be71beb4815eb1ae (table test-table [id=b19bded3347d49cb969115439dd0e0b5]) successfully deleted
I20260620 18:39:36.488592 19264 tablet_bootstrap.cc:492] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324: No bootstrap required, opened a new log
I20260620 18:39:36.488636 19264 ts_tablet_manager.cc:1397] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 18:39:36.488777 19264 raft_consensus.cc:348] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.488849 19264 raft_consensus.cc:374] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:36.488869 19264 raft_consensus.cc:729] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1beed494f744b6a28515dd3e65a324, State: Initialized, Role: FOLLOWER
I20260620 18:39:36.488915 19264 consensus_queue.cc:260] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.489002 19264 ts_tablet_manager.cc:1428] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:36.490651 19288 tablet_replica.cc:330] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb: stopping tablet replica
I20260620 18:39:36.490772 19288 raft_consensus.cc:2229] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:36.490944 19288 raft_consensus.cc:2258] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:36.491330 19288 ts_tablet_manager.cc:1905] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:36.492769 19288 ts_tablet_manager.cc:1918] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:36.492854 19288 log.cc:1196] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/wal/wals/d01b57881bea4199be71beb4815eb1ae
I20260620 18:39:36.493111 19288 ts_tablet_manager.cc:1939] T d01b57881bea4199be71beb4815eb1ae P 0ea5e748bc214b2a9f5829ab5f744ebb: Deleting consensus metadata
I20260620 18:39:36.493520 18739 catalog_manager.cc:4841] TS 0ea5e748bc214b2a9f5829ab5f744ebb (127.10.250.65:32873): tablet d01b57881bea4199be71beb4815eb1ae (table test-table [id=b19bded3347d49cb969115439dd0e0b5]) successfully deleted
W20260620 18:39:36.548154 18957 tablet.cc:2369] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:39:36.557032 19282 raft_consensus.cc:482] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:36.557163 19282 raft_consensus.cc:504] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.557415 19282 leader_election.cc:288] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bf4bbd6fdb54e81aedd96b25f7a6fed (127.10.250.66:40357), 6a1beed494f744b6a28515dd3e65a324 (127.10.250.67:33945)
I20260620 18:39:36.557616 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938e9fc88869444887e42458184b3c81" candidate_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" is_pre_election: true
I20260620 18:39:36.557658 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938e9fc88869444887e42458184b3c81" candidate_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1beed494f744b6a28515dd3e65a324" is_pre_election: true
I20260620 18:39:36.557729 19050 raft_consensus.cc:2454] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea5e748bc214b2a9f5829ab5f744ebb in term 0.
I20260620 18:39:36.557765 19190 raft_consensus.cc:2454] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea5e748bc214b2a9f5829ab5f744ebb in term 0.
I20260620 18:39:36.557922 18843 leader_election.cc:302] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea5e748bc214b2a9f5829ab5f744ebb, 9bf4bbd6fdb54e81aedd96b25f7a6fed; no voters:
I20260620 18:39:36.558058 19282 raft_consensus.cc:2781] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:36.558121 19282 raft_consensus.cc:482] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:36.558148 19282 raft_consensus.cc:3037] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:36.559005 19282 raft_consensus.cc:504] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.559156 19282 leader_election.cc:288] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [CANDIDATE]: Term 1 election: Requested vote from peers 9bf4bbd6fdb54e81aedd96b25f7a6fed (127.10.250.66:40357), 6a1beed494f744b6a28515dd3e65a324 (127.10.250.67:33945)
I20260620 18:39:36.559299 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938e9fc88869444887e42458184b3c81" candidate_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed"
I20260620 18:39:36.559410 19050 raft_consensus.cc:3037] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:36.559383 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938e9fc88869444887e42458184b3c81" candidate_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1beed494f744b6a28515dd3e65a324"
I20260620 18:39:36.559495 19190 raft_consensus.cc:3037] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:36.560097 19190 raft_consensus.cc:2454] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea5e748bc214b2a9f5829ab5f744ebb in term 1.
I20260620 18:39:36.560222 19050 raft_consensus.cc:2454] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea5e748bc214b2a9f5829ab5f744ebb in term 1.
I20260620 18:39:36.560271 18842 leader_election.cc:302] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea5e748bc214b2a9f5829ab5f744ebb, 6a1beed494f744b6a28515dd3e65a324; no voters:
I20260620 18:39:36.560359 19282 raft_consensus.cc:2781] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:36.560420 19282 raft_consensus.cc:686] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 LEADER]: Becoming Leader. State: Replica: 0ea5e748bc214b2a9f5829ab5f744ebb, State: Running, Role: LEADER
I20260620 18:39:36.560487 19282 consensus_queue.cc:237] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.561169 18753 catalog_manager.cc:5495] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ea5e748bc214b2a9f5829ab5f744ebb (127.10.250.65). New cstate: current_term: 1 leader_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } health_report { overall_health: UNKNOWN } } }
Jun 20 18:39:36 dist-test-slave-wp6g krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980775, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.126@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM
I20260620 18:39:36.603329 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.595396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33824 (local address 127.10.250.126:38757)
0620 18:39:36.595467 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.595470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.595685 (+ 215us) server_negotiation.cc:407] Connection header received
0620 18:39:36.595737 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.595737 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.595748 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.595805 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:36.596501 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.597025 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.597945 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.598128 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.598192 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.600593 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:36.600594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:36.600594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:36.600602 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:36.602125 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.602508 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.602508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.602509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.602615 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.602875 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.602876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.602877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.602997 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:36.603059 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.603180 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.603257 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 18:39:36.603513 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.595353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38757 (local address 127.0.0.1:33824)
0620 18:39:36.595578 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.595592 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.595674 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.595845 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.595847 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.596278 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:36.596346 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.596350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.597114 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.597116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.597853 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.597856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.598209 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.598210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.598285 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.598900 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:36.598906 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:36.600492 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:36.602216 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:36.602224 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:36.602226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:36.602435 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:36.602683 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:36.602684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:36.602684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:36.602752 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:36.603061 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:36.603067 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:36.603110 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.603162 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.603321 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":72,"thread_start_us":84,"threads_started":1}
I20260620 18:39:36.605522 18753 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:33824:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:39:36.605727 18753 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:36.607224 18890 tablet_service.cc:1467] Processing CreateTablet for tablet aac067bd58554807b259e97a22d4e263 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=623d2265081d421583684e497496e9e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:39:36.607245 19170 tablet_service.cc:1467] Processing CreateTablet for tablet aac067bd58554807b259e97a22d4e263 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=623d2265081d421583684e497496e9e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:39:36.607271 19030 tablet_service.cc:1467] Processing CreateTablet for tablet aac067bd58554807b259e97a22d4e263 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=623d2265081d421583684e497496e9e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:39:36.607398 18890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aac067bd58554807b259e97a22d4e263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:36.607398 19030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aac067bd58554807b259e97a22d4e263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:36.607417 19170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aac067bd58554807b259e97a22d4e263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:36.608589 19264 tablet_bootstrap.cc:492] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324: Bootstrap starting.
I20260620 18:39:36.608865 19266 tablet_bootstrap.cc:492] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Bootstrap starting.
I20260620 18:39:36.608872 19265 tablet_bootstrap.cc:492] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb: Bootstrap starting.
I20260620 18:39:36.609050 19264 tablet_bootstrap.cc:654] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:36.609176 19265 tablet_bootstrap.cc:654] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:36.609314 19266 tablet_bootstrap.cc:654] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:36.609831 19264 tablet_bootstrap.cc:492] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324: No bootstrap required, opened a new log
I20260620 18:39:36.609889 19264 ts_tablet_manager.cc:1397] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:36.610028 19266 tablet_bootstrap.cc:492] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: No bootstrap required, opened a new log
I20260620 18:39:36.610083 19265 tablet_bootstrap.cc:492] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb: No bootstrap required, opened a new log
I20260620 18:39:36.610085 19266 ts_tablet_manager.cc:1397] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:36.610049 19264 raft_consensus.cc:348] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.610123 19265 ts_tablet_manager.cc:1397] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:36.610141 19264 raft_consensus.cc:374] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:36.610172 19264 raft_consensus.cc:729] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1beed494f744b6a28515dd3e65a324, State: Initialized, Role: FOLLOWER
I20260620 18:39:36.610217 19264 consensus_queue.cc:260] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.610239 19265 raft_consensus.cc:348] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.610230 19266 raft_consensus.cc:348] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.610297 19266 raft_consensus.cc:374] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:36.610297 19265 raft_consensus.cc:374] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:36.610324 19265 raft_consensus.cc:729] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea5e748bc214b2a9f5829ab5f744ebb, State: Initialized, Role: FOLLOWER
I20260620 18:39:36.610324 19266 raft_consensus.cc:729] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf4bbd6fdb54e81aedd96b25f7a6fed, State: Initialized, Role: FOLLOWER
I20260620 18:39:36.610322 19264 ts_tablet_manager.cc:1428] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:36.610368 19265 consensus_queue.cc:260] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.610368 19266 consensus_queue.cc:260] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.610458 19266 ts_tablet_manager.cc:1428] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:36.610463 19265 ts_tablet_manager.cc:1428] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:36.610482 19270 tablet_replica.cc:439] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } }
I20260620 18:39:36.610611 19270 tablet_replica.cc:442] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:36.610598 19272 tablet_replica.cc:439] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } }
I20260620 18:39:36.610692 19272 tablet_replica.cc:442] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:36.610894 19271 tablet_replica.cc:439] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } }
I20260620 18:39:36.610998 19271 tablet_replica.cc:442] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 18:39:36.615509 19237 tablet.cc:2369] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:39:36.740921 19270 raft_consensus.cc:482] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:36.741024 19270 raft_consensus.cc:504] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.741436 19270 leader_election.cc:288] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ea5e748bc214b2a9f5829ab5f744ebb (127.10.250.65:32873), 9bf4bbd6fdb54e81aedd96b25f7a6fed (127.10.250.66:40357)
I20260620 18:39:36.744867 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.741639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:32873 (local address 127.10.250.67:51149)
0620 18:39:36.741699 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.741712 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.741802 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.741966 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.741967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.741969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:36.742048 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.742050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.742874 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.742875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.743996 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.743999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.744672 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.744673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.744732 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.744735 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.744777 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.744810 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 18:39:36.744948 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.741739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:51149 (local address 127.10.250.65:32873)
0620 18:39:36.741796 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.741798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.741812 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:39:36.741868 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.741869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.741882 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.741942 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:36.742136 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.742773 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.744092 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.744596 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.744669 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.744678 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.744807 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.744849 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 18:39:36.745184 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac067bd58554807b259e97a22d4e263" candidate_uuid: "6a1beed494f744b6a28515dd3e65a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" is_pre_election: true
I20260620 18:39:36.745276 19303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.741750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:40357 (local address 127.10.250.67:45257)
0620 18:39:36.742006 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.742017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.742075 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.742214 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.742215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.742216 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:36.742542 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.742544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.743233 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.743234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.744254 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.744256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.745005 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.745006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.745096 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.745099 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.745174 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.745217 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":109,"threads_started":1}
I20260620 18:39:36.745324 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.741837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:45257 (local address 127.10.250.66:40357)
0620 18:39:36.741944 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.741946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.742082 (+ 136us) server_negotiation.cc:407] Connection header received
0620 18:39:36.742134 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.742135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.742145 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.742200 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:36.742617 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.743157 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.744328 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.744869 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.744939 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.744947 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.745201 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.745246 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:39:36.745338 18910 raft_consensus.cc:2454] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a1beed494f744b6a28515dd3e65a324 in term 0.
I20260620 18:39:36.745572 19123 leader_election.cc:302] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea5e748bc214b2a9f5829ab5f744ebb, 6a1beed494f744b6a28515dd3e65a324; no voters:
I20260620 18:39:36.745709 19270 raft_consensus.cc:2781] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:36.745755 19270 raft_consensus.cc:482] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:36.745783 19270 raft_consensus.cc:3037] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:36.745844 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac067bd58554807b259e97a22d4e263" candidate_uuid: "6a1beed494f744b6a28515dd3e65a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" is_pre_election: true
I20260620 18:39:36.745935 19050 raft_consensus.cc:2454] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a1beed494f744b6a28515dd3e65a324 in term 0.
I20260620 18:39:36.746622 19270 raft_consensus.cc:504] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.746752 19270 leader_election.cc:288] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [CANDIDATE]: Term 1 election: Requested vote from peers 0ea5e748bc214b2a9f5829ab5f744ebb (127.10.250.65:32873), 9bf4bbd6fdb54e81aedd96b25f7a6fed (127.10.250.66:40357)
I20260620 18:39:36.746868 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac067bd58554807b259e97a22d4e263" candidate_uuid: "6a1beed494f744b6a28515dd3e65a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb"
I20260620 18:39:36.746939 18910 raft_consensus.cc:3037] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:36.746913 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac067bd58554807b259e97a22d4e263" candidate_uuid: "6a1beed494f744b6a28515dd3e65a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed"
I20260620 18:39:36.746981 19050 raft_consensus.cc:3037] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:36.747702 19050 raft_consensus.cc:2454] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a1beed494f744b6a28515dd3e65a324 in term 1.
I20260620 18:39:36.747763 18910 raft_consensus.cc:2454] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a1beed494f744b6a28515dd3e65a324 in term 1.
I20260620 18:39:36.747888 19123 leader_election.cc:302] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1beed494f744b6a28515dd3e65a324, 9bf4bbd6fdb54e81aedd96b25f7a6fed; no voters:
I20260620 18:39:36.748004 19270 raft_consensus.cc:2781] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:36.748176 19270 raft_consensus.cc:686] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [term 1 LEADER]: Becoming Leader. State: Replica: 6a1beed494f744b6a28515dd3e65a324, State: Running, Role: LEADER
I20260620 18:39:36.748263 19270 consensus_queue.cc:237] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.748668 19304 tablet_replica.cc:439] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324: TxnStatusTablet state changed. Reason: New leader 6a1beed494f744b6a28515dd3e65a324. Latest consensus state: current_term: 1 leader_uuid: "6a1beed494f744b6a28515dd3e65a324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } }
I20260620 18:39:36.748762 19304 tablet_replica.cc:442] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:39:36.748953 19306 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:39:36.749074 19306 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:39:36.749123 18753 catalog_manager.cc:5495] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a1beed494f744b6a28515dd3e65a324 (127.10.250.67). New cstate: current_term: 1 leader_uuid: "6a1beed494f744b6a28515dd3e65a324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } health_report { overall_health: HEALTHY } } }
I20260620 18:39:36.758086 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.755444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:33945 (local address 127.0.0.1:43894)
0620 18:39:36.755531 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.755560 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.755609 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.755807 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.755808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.755809 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:39:36.755861 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.755864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.756553 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.756554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.757134 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.757136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.757521 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.757522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.757592 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.757615 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:39:36.757904 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:39:36.757915 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.757960 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.758016 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 18:39:36.758069 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.755516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.10.250.67:33945)
0620 18:39:36.755586 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.755588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.755598 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:39:36.755694 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.755695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.755712 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.755802 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:36.755941 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.756459 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.757222 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.757431 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.757518 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.757712 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:36.757820 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:36.757917 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.757973 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.757993 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:39:36.759373 19050 raft_consensus.cc:1261] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Refusing update from remote peer 6a1beed494f744b6a28515dd3e65a324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:36.759413 18910 raft_consensus.cc:1261] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 FOLLOWER]: Refusing update from remote peer 6a1beed494f744b6a28515dd3e65a324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:36.759730 19304 consensus_queue.cc:1035] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:36.759898 19270 consensus_queue.cc:1035] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:36.760567 19272 tablet_replica.cc:439] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb: TxnStatusTablet state changed. Reason: New leader 6a1beed494f744b6a28515dd3e65a324. Latest consensus state: current_term: 1 leader_uuid: "6a1beed494f744b6a28515dd3e65a324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } }
I20260620 18:39:36.760661 19272 tablet_replica.cc:442] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:36.760723 19271 tablet_replica.cc:439] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: TxnStatusTablet state changed. Reason: New leader 6a1beed494f744b6a28515dd3e65a324. Latest consensus state: current_term: 1 leader_uuid: "6a1beed494f744b6a28515dd3e65a324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } }
I20260620 18:39:36.760816 19271 tablet_replica.cc:442] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:36.761180 19304 tablet_replica.cc:439] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a1beed494f744b6a28515dd3e65a324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } }
I20260620 18:39:36.761255 19304 tablet_replica.cc:442] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:39:36.761384 19271 tablet_replica.cc:439] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a1beed494f744b6a28515dd3e65a324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } }
I20260620 18:39:36.761466 19271 tablet_replica.cc:442] T aac067bd58554807b259e97a22d4e263 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:36.761900 19304 tablet_replica.cc:439] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a1beed494f744b6a28515dd3e65a324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } }
I20260620 18:39:36.761974 19304 tablet_replica.cc:442] T aac067bd58554807b259e97a22d4e263 P 6a1beed494f744b6a28515dd3e65a324: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:39:36.762260 19277 mvcc.cc:205] Tried to move back new op lower bound from 7298993261604089856 to 7298993261561647104. Current Snapshot: MvccSnapshot[applied={T|T < 7298993261604089856}]
I20260620 18:39:36.762374 19272 tablet_replica.cc:439] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a1beed494f744b6a28515dd3e65a324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } } }
I20260620 18:39:36.762444 19272 tablet_replica.cc:442] T aac067bd58554807b259e97a22d4e263 P 0ea5e748bc214b2a9f5829ab5f744ebb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:39:36.768602 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.765681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:39289 (local address 127.10.250.67:33945)
0620 18:39:36.765723 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.765726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.765860 (+ 134us) server_negotiation.cc:407] Connection header received
0620 18:39:36.766001 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.766002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.766012 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.766104 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:36.766196 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.767038 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.767812 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.768114 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.768187 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.768351 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:36.768385 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:36.768443 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.768488 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.768509 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:39:36.768584 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.765629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:33945 (local address 127.10.250.65:39289)
0620 18:39:36.765787 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.765799 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.765876 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.766106 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.766107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.766109 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:39:36.766154 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.766156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.767118 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.767119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.767733 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.767735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.768192 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.768192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.768235 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.768249 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:39:36.768449 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:39:36.768451 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.768485 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.768525 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":94,"threads_started":1}
I20260620 18:39:36.771364 19050 raft_consensus.cc:1261] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Refusing update from remote peer 0ea5e748bc214b2a9f5829ab5f744ebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:36.771392 19190 raft_consensus.cc:1261] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324 [term 1 FOLLOWER]: Refusing update from remote peer 0ea5e748bc214b2a9f5829ab5f744ebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:36.771574 19272 consensus_queue.cc:1035] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:36.771660 19282 consensus_queue.cc:1035] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:36.779728 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.776831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:32873 (local address 127.10.250.67:46045)
0620 18:39:36.776891 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:36.776912 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:36.776970 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:36.777084 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:36.777085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:36.777087 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:39:36.777151 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.777152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.777743 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.777744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.778697 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:36.778700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.779129 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.779130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:36.779225 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.779247 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:39:36.779548 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:39:36.779550 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:39:36.779609 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 18:39:36.779652 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 18:39:36.779723 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.776876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:46045 (local address 127.10.250.65:32873)
0620 18:39:36.776920 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.776922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.776950 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:39:36.777010 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.777010 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.777018 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.777062 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:36.777212 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.777669 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.778814 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.779047 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.779101 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.779427 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:36.779469 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:36.779527 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.779611 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.779631 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:39:36.788232 18752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:36.788362 18752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:36.788988 18752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733: Sending DeleteTablet for 3 replicas of tablet 938e9fc88869444887e42458184b3c81
I20260620 18:39:36.789260 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet 938e9fc88869444887e42458184b3c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:36 UTC) from {username='oryx', principal='oryx/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:43886
I20260620 18:39:36.789363 19287 tablet_replica.cc:330] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324: stopping tablet replica
I20260620 18:39:36.789354 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 938e9fc88869444887e42458184b3c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:36 UTC) from {username='oryx', principal='oryx/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:37198
I20260620 18:39:36.789419 19287 raft_consensus.cc:2229] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:36.789378 19030 tablet_service.cc:1514] Processing DeleteTablet for tablet 938e9fc88869444887e42458184b3c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:36 UTC) from {username='oryx', principal='oryx/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:51934
I20260620 18:39:36.789455 19287 raft_consensus.cc:2258] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:36.789496 19288 tablet_replica.cc:330] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb: stopping tablet replica
I20260620 18:39:36.789497 19289 tablet_replica.cc:330] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: stopping tablet replica
I20260620 18:39:36.789557 19289 raft_consensus.cc:2229] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:36.789561 19288 raft_consensus.cc:2229] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:36.789598 19289 raft_consensus.cc:2258] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:36.789619 19288 raft_consensus.cc:2258] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:36.789932 19288 ts_tablet_manager.cc:1905] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:36.790153 19287 ts_tablet_manager.cc:1905] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:36.792110 19288 ts_tablet_manager.cc:1918] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:36.792151 19287 ts_tablet_manager.cc:1918] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:36.792177 19288 log.cc:1196] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/wal/wals/938e9fc88869444887e42458184b3c81
I20260620 18:39:36.792209 19287 log.cc:1196] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/938e9fc88869444887e42458184b3c81
I20260620 18:39:36.792456 19287 ts_tablet_manager.cc:1939] T 938e9fc88869444887e42458184b3c81 P 6a1beed494f744b6a28515dd3e65a324: Deleting consensus metadata
I20260620 18:39:36.792469 19288 ts_tablet_manager.cc:1939] T 938e9fc88869444887e42458184b3c81 P 0ea5e748bc214b2a9f5829ab5f744ebb: Deleting consensus metadata
I20260620 18:39:36.792811 18739 catalog_manager.cc:4841] TS 0ea5e748bc214b2a9f5829ab5f744ebb (127.10.250.65:32873): tablet 938e9fc88869444887e42458184b3c81 (table test-table [id=61686d053de240a9968a0e78493b07d5]) successfully deleted
I20260620 18:39:36.792814 18738 catalog_manager.cc:4841] TS 6a1beed494f744b6a28515dd3e65a324 (127.10.250.67:33945): tablet 938e9fc88869444887e42458184b3c81 (table test-table [id=61686d053de240a9968a0e78493b07d5]) successfully deleted
I20260620 18:39:36.794467 19289 ts_tablet_manager.cc:1905] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:36.796250 19289 ts_tablet_manager.cc:1918] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:36.796341 19289 log.cc:1196] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/938e9fc88869444887e42458184b3c81
I20260620 18:39:36.796559 19289 ts_tablet_manager.cc:1939] T 938e9fc88869444887e42458184b3c81 P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Deleting consensus metadata
I20260620 18:39:36.796922 18739 catalog_manager.cc:4841] TS 9bf4bbd6fdb54e81aedd96b25f7a6fed (127.10.250.66:40357): tablet 938e9fc88869444887e42458184b3c81 (table test-table [id=61686d053de240a9968a0e78493b07d5]) successfully deleted
Jun 20 18:39:36 dist-test-slave-wp6g krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:39:36 dist-test-slave-wp6g krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:36 dist-test-slave-wp6g krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:39:36.814797 18819 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:36.808567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33830 (local address 127.10.250.126:38757)
0620 18:39:36.808638 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.808641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.808791 (+ 150us) server_negotiation.cc:407] Connection header received
0620 18:39:36.808844 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.808844 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.808856 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.808930 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:36.809584 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.810153 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.811011 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.811269 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.811350 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.814708 (+ 3358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33830: BlockingRecv error: recv got EOF from 127.0.0.1:33830 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 18:39:36 dist-test-slave-wp6g krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM
I20260620 18:39:36.823427 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.815733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.10.250.126:38757)
0620 18:39:36.815811 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.815813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.815875 (+ 62us) server_negotiation.cc:407] Connection header received
0620 18:39:36.815963 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.815964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.815977 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.816053 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:36.816656 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.817157 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.817939 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.818140 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.818239 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.820537 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:36.820538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:36.820539 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:36.820547 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:36.822170 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.822614 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.822616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.822616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.822681 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:36.822881 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:36.822882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:36.822882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:36.823007 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:36.823117 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.823229 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.823313 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:39:36.825083 18752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:36.825244 18752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:36.826951 19030 tablet_service.cc:1467] Processing CreateTablet for tablet aaaecce320af483783833fe8c31caf3b (DEFAULT_TABLE table=test-table [id=625cbc1200bf45d2aaa53b9044233340]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:36.827028 18890 tablet_service.cc:1467] Processing CreateTablet for tablet aaaecce320af483783833fe8c31caf3b (DEFAULT_TABLE table=test-table [id=625cbc1200bf45d2aaa53b9044233340]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:36.827100 19030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaaecce320af483783833fe8c31caf3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:36.827143 18890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaaecce320af483783833fe8c31caf3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:36.828285 19265 tablet_bootstrap.cc:492] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb: Bootstrap starting.
I20260620 18:39:36.828524 19170 tablet_service.cc:1467] Processing CreateTablet for tablet aaaecce320af483783833fe8c31caf3b (DEFAULT_TABLE table=test-table [id=625cbc1200bf45d2aaa53b9044233340]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:36.828564 19265 tablet_bootstrap.cc:654] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:36.828644 19170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaaecce320af483783833fe8c31caf3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:36.829063 19265 tablet_bootstrap.cc:492] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb: No bootstrap required, opened a new log
I20260620 18:39:36.829097 19265 ts_tablet_manager.cc:1397] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 18:39:36.829180 19265 raft_consensus.cc:348] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.829226 19265 raft_consensus.cc:374] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:36.829239 19265 raft_consensus.cc:729] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea5e748bc214b2a9f5829ab5f744ebb, State: Initialized, Role: FOLLOWER
I20260620 18:39:36.829303 19265 consensus_queue.cc:260] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.829414 19265 ts_tablet_manager.cc:1428] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:36.829931 19264 tablet_bootstrap.cc:492] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324: Bootstrap starting.
I20260620 18:39:36.829964 19266 tablet_bootstrap.cc:492] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Bootstrap starting.
I20260620 18:39:36.830245 19264 tablet_bootstrap.cc:654] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:36.830389 19266 tablet_bootstrap.cc:654] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:36.830907 19264 tablet_bootstrap.cc:492] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324: No bootstrap required, opened a new log
I20260620 18:39:36.830955 19264 ts_tablet_manager.cc:1397] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:36.831118 19264 raft_consensus.cc:348] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.831197 19264 raft_consensus.cc:374] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:36.831223 19264 raft_consensus.cc:729] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1beed494f744b6a28515dd3e65a324, State: Initialized, Role: FOLLOWER
I20260620 18:39:36.831274 19264 consensus_queue.cc:260] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.831398 19264 ts_tablet_manager.cc:1428] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:36.831912 19266 tablet_bootstrap.cc:492] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed: No bootstrap required, opened a new log
I20260620 18:39:36.831964 19266 ts_tablet_manager.cc:1397] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:39:36.832111 19266 raft_consensus.cc:348] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.832201 19266 raft_consensus.cc:374] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:36.832221 19266 raft_consensus.cc:729] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf4bbd6fdb54e81aedd96b25f7a6fed, State: Initialized, Role: FOLLOWER
I20260620 18:39:36.832270 19266 consensus_queue.cc:260] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.832350 19266 ts_tablet_manager.cc:1428] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:36.932003 19314 raft_consensus.cc:482] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:36.932113 19314 raft_consensus.cc:504] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.932243 19314 leader_election.cc:288] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ea5e748bc214b2a9f5829ab5f744ebb (127.10.250.65:32873), 9bf4bbd6fdb54e81aedd96b25f7a6fed (127.10.250.66:40357)
I20260620 18:39:36.932556 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaaecce320af483783833fe8c31caf3b" candidate_uuid: "6a1beed494f744b6a28515dd3e65a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" is_pre_election: true
I20260620 18:39:36.932538 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaaecce320af483783833fe8c31caf3b" candidate_uuid: "6a1beed494f744b6a28515dd3e65a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" is_pre_election: true
I20260620 18:39:36.932643 19050 raft_consensus.cc:2454] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a1beed494f744b6a28515dd3e65a324 in term 0.
I20260620 18:39:36.932663 18910 raft_consensus.cc:2454] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a1beed494f744b6a28515dd3e65a324 in term 0.
I20260620 18:39:36.932814 19123 leader_election.cc:302] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1beed494f744b6a28515dd3e65a324, 9bf4bbd6fdb54e81aedd96b25f7a6fed; no voters:
I20260620 18:39:36.932934 19314 raft_consensus.cc:2781] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:36.932981 19314 raft_consensus.cc:482] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:36.932996 19314 raft_consensus.cc:3037] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:36.933724 19314 raft_consensus.cc:504] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.933831 19314 leader_election.cc:288] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [CANDIDATE]: Term 1 election: Requested vote from peers 0ea5e748bc214b2a9f5829ab5f744ebb (127.10.250.65:32873), 9bf4bbd6fdb54e81aedd96b25f7a6fed (127.10.250.66:40357)
I20260620 18:39:36.934065 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaaecce320af483783833fe8c31caf3b" candidate_uuid: "6a1beed494f744b6a28515dd3e65a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb"
I20260620 18:39:36.934087 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaaecce320af483783833fe8c31caf3b" candidate_uuid: "6a1beed494f744b6a28515dd3e65a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed"
I20260620 18:39:36.934163 18910 raft_consensus.cc:3037] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:36.934175 19050 raft_consensus.cc:3037] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:36.934819 19050 raft_consensus.cc:2454] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a1beed494f744b6a28515dd3e65a324 in term 1.
I20260620 18:39:36.934957 18910 raft_consensus.cc:2454] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a1beed494f744b6a28515dd3e65a324 in term 1.
I20260620 18:39:36.935004 19123 leader_election.cc:302] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1beed494f744b6a28515dd3e65a324, 9bf4bbd6fdb54e81aedd96b25f7a6fed; no voters:
I20260620 18:39:36.935112 19314 raft_consensus.cc:2781] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:36.935169 19314 raft_consensus.cc:686] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [term 1 LEADER]: Becoming Leader. State: Replica: 6a1beed494f744b6a28515dd3e65a324, State: Running, Role: LEADER
I20260620 18:39:36.935220 19314 consensus_queue.cc:237] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.935781 18752 catalog_manager.cc:5495] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a1beed494f744b6a28515dd3e65a324 (127.10.250.67). New cstate: current_term: 1 leader_uuid: "6a1beed494f744b6a28515dd3e65a324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } health_report { overall_health: HEALTHY } } }
I20260620 18:39:36.948026 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:36.945620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.10.250.67:33945)
0620 18:39:36.945702 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:36.945705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:36.945715 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:39:36.945785 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:36.945786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:36.945800 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:36.945866 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:36.946017 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.946597 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.947351 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:36.947542 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:36.947615 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:36.947762 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:36.947800 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:36.947857 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:36.947918 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 18:39:36.947940 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:39:36.949004 18910 raft_consensus.cc:1261] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 FOLLOWER]: Refusing update from remote peer 6a1beed494f744b6a28515dd3e65a324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:36.949002 19050 raft_consensus.cc:1261] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Refusing update from remote peer 6a1beed494f744b6a28515dd3e65a324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:36.949270 19304 consensus_queue.cc:1035] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:36.949426 19314 consensus_queue.cc:1035] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:36.952652 18752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:36.952749 18752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:36.953419 18752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733: Sending DeleteTablet for 3 replicas of tablet aaaecce320af483783833fe8c31caf3b
I20260620 18:39:36.953663 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet aaaecce320af483783833fe8c31caf3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:36 UTC) from {username='oryx', principal='oryx/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:43886
I20260620 18:39:36.953653 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet aaaecce320af483783833fe8c31caf3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:36 UTC) from {username='oryx', principal='oryx/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:37198
I20260620 18:39:36.953753 19287 tablet_replica.cc:330] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324: stopping tablet replica
I20260620 18:39:36.953756 19030 tablet_service.cc:1514] Processing DeleteTablet for tablet aaaecce320af483783833fe8c31caf3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:36 UTC) from {username='oryx', principal='oryx/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:51934
I20260620 18:39:36.953800 19287 raft_consensus.cc:2229] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:36.953857 19289 tablet_replica.cc:330] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed: stopping tablet replica
I20260620 18:39:36.953909 19287 raft_consensus.cc:2258] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:36.953920 19289 raft_consensus.cc:2229] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:36.953971 19289 raft_consensus.cc:2258] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:36.953996 19288 tablet_replica.cc:330] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb: stopping tablet replica
I20260620 18:39:36.954054 19288 raft_consensus.cc:2229] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:36.954107 19288 raft_consensus.cc:2258] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:36.954388 19289 ts_tablet_manager.cc:1905] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:36.954470 19287 ts_tablet_manager.cc:1905] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:36.956148 19287 ts_tablet_manager.cc:1918] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:36.956177 19288 ts_tablet_manager.cc:1905] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:36.956189 19289 ts_tablet_manager.cc:1918] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:36.956213 19287 log.cc:1196] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/aaaecce320af483783833fe8c31caf3b
I20260620 18:39:36.956240 19289 log.cc:1196] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/aaaecce320af483783833fe8c31caf3b
I20260620 18:39:36.956162 18752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:36.956317 18752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:36.956456 19287 ts_tablet_manager.cc:1939] T aaaecce320af483783833fe8c31caf3b P 6a1beed494f744b6a28515dd3e65a324: Deleting consensus metadata
I20260620 18:39:36.956501 19289 ts_tablet_manager.cc:1939] T aaaecce320af483783833fe8c31caf3b P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Deleting consensus metadata
I20260620 18:39:36.956808 18739 catalog_manager.cc:4841] TS 9bf4bbd6fdb54e81aedd96b25f7a6fed (127.10.250.66:40357): tablet aaaecce320af483783833fe8c31caf3b (table test-table [id=625cbc1200bf45d2aaa53b9044233340]) successfully deleted
I20260620 18:39:36.957166 18738 catalog_manager.cc:4841] TS 6a1beed494f744b6a28515dd3e65a324 (127.10.250.67:33945): tablet aaaecce320af483783833fe8c31caf3b (table test-table [id=625cbc1200bf45d2aaa53b9044233340]) successfully deleted
I20260620 18:39:36.957705 19288 ts_tablet_manager.cc:1918] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:36.957765 19288 log.cc:1196] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-0/wal/wals/aaaecce320af483783833fe8c31caf3b
I20260620 18:39:36.958000 19288 ts_tablet_manager.cc:1939] T aaaecce320af483783833fe8c31caf3b P 0ea5e748bc214b2a9f5829ab5f744ebb: Deleting consensus metadata
I20260620 18:39:36.958060 19030 tablet_service.cc:1467] Processing CreateTablet for tablet 3d29a043334c4e55b2aea5b9f91d997a (DEFAULT_TABLE table=test-table [id=791e1f33b2e246f78968eb868c4a4988]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:36.958065 18890 tablet_service.cc:1467] Processing CreateTablet for tablet 3d29a043334c4e55b2aea5b9f91d997a (DEFAULT_TABLE table=test-table [id=791e1f33b2e246f78968eb868c4a4988]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:36.958153 19030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d29a043334c4e55b2aea5b9f91d997a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:36.958171 18890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d29a043334c4e55b2aea5b9f91d997a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:36.959203 19266 tablet_bootstrap.cc:492] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Bootstrap starting.
I20260620 18:39:36.959376 19265 tablet_bootstrap.cc:492] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb: Bootstrap starting.
I20260620 18:39:36.959669 19265 tablet_bootstrap.cc:654] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:36.959823 19266 tablet_bootstrap.cc:654] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:36.959928 19170 tablet_service.cc:1467] Processing CreateTablet for tablet 3d29a043334c4e55b2aea5b9f91d997a (DEFAULT_TABLE table=test-table [id=791e1f33b2e246f78968eb868c4a4988]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:36.960021 19170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d29a043334c4e55b2aea5b9f91d997a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:36.960407 19266 tablet_bootstrap.cc:492] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed: No bootstrap required, opened a new log
I20260620 18:39:36.960438 19266 ts_tablet_manager.cc:1397] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:36.960541 19266 raft_consensus.cc:348] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.960587 19266 raft_consensus.cc:374] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:36.960601 19266 raft_consensus.cc:729] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf4bbd6fdb54e81aedd96b25f7a6fed, State: Initialized, Role: FOLLOWER
I20260620 18:39:36.960659 19266 consensus_queue.cc:260] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.960753 19266 ts_tablet_manager.cc:1428] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:36.961206 19264 tablet_bootstrap.cc:492] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324: Bootstrap starting.
I20260620 18:39:36.961534 18739 catalog_manager.cc:4841] TS 0ea5e748bc214b2a9f5829ab5f744ebb (127.10.250.65:32873): tablet aaaecce320af483783833fe8c31caf3b (table test-table [id=625cbc1200bf45d2aaa53b9044233340]) successfully deleted
I20260620 18:39:36.961565 19264 tablet_bootstrap.cc:654] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:36.961912 19265 tablet_bootstrap.cc:492] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb: No bootstrap required, opened a new log
I20260620 18:39:36.961961 19265 ts_tablet_manager.cc:1397] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 18:39:36.962077 19264 tablet_bootstrap.cc:492] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324: No bootstrap required, opened a new log
I20260620 18:39:36.962065 19265 raft_consensus.cc:348] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.962110 19264 ts_tablet_manager.cc:1397] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:36.962127 19265 raft_consensus.cc:374] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:36.962147 19265 raft_consensus.cc:729] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea5e748bc214b2a9f5829ab5f744ebb, State: Initialized, Role: FOLLOWER
I20260620 18:39:36.962195 19265 consensus_queue.cc:260] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.962229 19264 raft_consensus.cc:348] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.962275 19265 ts_tablet_manager.cc:1428] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:36.962298 19264 raft_consensus.cc:374] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:36.962319 19264 raft_consensus.cc:729] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1beed494f744b6a28515dd3e65a324, State: Initialized, Role: FOLLOWER
I20260620 18:39:36.962358 19264 consensus_queue.cc:260] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:36.962459 19264 ts_tablet_manager.cc:1428] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:37.019850 19304 raft_consensus.cc:482] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:37.019965 19304 raft_consensus.cc:504] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:37.020124 19304 leader_election.cc:288] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ea5e748bc214b2a9f5829ab5f744ebb (127.10.250.65:32873), 9bf4bbd6fdb54e81aedd96b25f7a6fed (127.10.250.66:40357)
I20260620 18:39:37.020325 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d29a043334c4e55b2aea5b9f91d997a" candidate_uuid: "6a1beed494f744b6a28515dd3e65a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" is_pre_election: true
I20260620 18:39:37.020367 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d29a043334c4e55b2aea5b9f91d997a" candidate_uuid: "6a1beed494f744b6a28515dd3e65a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" is_pre_election: true
I20260620 18:39:37.020442 18910 raft_consensus.cc:2454] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a1beed494f744b6a28515dd3e65a324 in term 0.
I20260620 18:39:37.020465 19050 raft_consensus.cc:2454] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a1beed494f744b6a28515dd3e65a324 in term 0.
I20260620 18:39:37.020673 19123 leader_election.cc:302] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1beed494f744b6a28515dd3e65a324, 9bf4bbd6fdb54e81aedd96b25f7a6fed; no voters:
I20260620 18:39:37.020784 19304 raft_consensus.cc:2781] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:37.020852 19304 raft_consensus.cc:482] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:37.020884 19304 raft_consensus.cc:3037] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:37.021551 19304 raft_consensus.cc:504] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:37.021680 19304 leader_election.cc:288] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [CANDIDATE]: Term 1 election: Requested vote from peers 0ea5e748bc214b2a9f5829ab5f744ebb (127.10.250.65:32873), 9bf4bbd6fdb54e81aedd96b25f7a6fed (127.10.250.66:40357)
I20260620 18:39:37.021883 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d29a043334c4e55b2aea5b9f91d997a" candidate_uuid: "6a1beed494f744b6a28515dd3e65a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb"
I20260620 18:39:37.021967 18910 raft_consensus.cc:3037] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:37.022625 18910 raft_consensus.cc:2454] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a1beed494f744b6a28515dd3e65a324 in term 1.
I20260620 18:39:37.022603 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d29a043334c4e55b2aea5b9f91d997a" candidate_uuid: "6a1beed494f744b6a28515dd3e65a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed"
I20260620 18:39:37.022683 19050 raft_consensus.cc:3037] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:37.023511 19050 raft_consensus.cc:2454] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a1beed494f744b6a28515dd3e65a324 in term 1.
I20260620 18:39:37.023764 19123 leader_election.cc:302] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1beed494f744b6a28515dd3e65a324, 9bf4bbd6fdb54e81aedd96b25f7a6fed; no voters:
I20260620 18:39:37.023874 19304 raft_consensus.cc:2781] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:37.023944 19304 raft_consensus.cc:686] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [term 1 LEADER]: Becoming Leader. State: Replica: 6a1beed494f744b6a28515dd3e65a324, State: Running, Role: LEADER
I20260620 18:39:37.024001 19304 consensus_queue.cc:237] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } }
I20260620 18:39:37.024633 18751 catalog_manager.cc:5495] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a1beed494f744b6a28515dd3e65a324 (127.10.250.67). New cstate: current_term: 1 leader_uuid: "6a1beed494f744b6a28515dd3e65a324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a1beed494f744b6a28515dd3e65a324" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33945 } health_report { overall_health: HEALTHY } } }
I20260620 18:39:37.042016 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:37.039469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:33945 (local address 127.10.250.67:41699)
0620 18:39:37.039522 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:37.039550 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:37.039590 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:37.039726 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:37.039726 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:37.039727 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:39:37.039770 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:37.039771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:37.040658 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:37.040659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:37.041209 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:37.041210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:37.041648 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:37.041649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:37.041691 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:37.041696 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:39:37.041850 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:39:37.041851 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 18:39:37.041883 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 18:39:37.041921 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":2}
I20260620 18:39:37.042021 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:37.039537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:41699 (local address 127.10.250.67:33945)
0620 18:39:37.039577 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:37.039579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:37.039587 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:39:37.039650 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:37.039651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:37.039661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:37.039724 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:37.039831 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:37.040539 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:37.041289 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:37.041565 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:37.041635 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:37.041761 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:37.041794 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:37.041845 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:37.041894 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 18:39:37.041918 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":20}
I20260620 18:39:37.044349 18910 raft_consensus.cc:1261] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 FOLLOWER]: Refusing update from remote peer 6a1beed494f744b6a28515dd3e65a324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:37.044391 19050 raft_consensus.cc:1261] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Refusing update from remote peer 6a1beed494f744b6a28515dd3e65a324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:37.044569 19304 consensus_queue.cc:1035] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea5e748bc214b2a9f5829ab5f744ebb" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 32873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:37.044647 19314 consensus_queue.cc:1035] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bf4bbd6fdb54e81aedd96b25f7a6fed" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:37.055729 18751 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:37.055866 18751 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:37.056542 18751 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d888330b95741c992e8adad4ad3f733: Sending DeleteTablet for 3 replicas of tablet 3d29a043334c4e55b2aea5b9f91d997a
I20260620 18:39:37.056813 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d29a043334c4e55b2aea5b9f91d997a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:37 UTC) from {username='oryx', principal='oryx/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:43886
I20260620 18:39:37.056830 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d29a043334c4e55b2aea5b9f91d997a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:37 UTC) from {username='oryx', principal='oryx/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:37198
I20260620 18:39:37.056931 19288 tablet_replica.cc:330] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb: stopping tablet replica
I20260620 18:39:37.056951 19287 tablet_replica.cc:330] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324: stopping tablet replica
I20260620 18:39:37.056977 19288 raft_consensus.cc:2229] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:37.057008 19288 raft_consensus.cc:2258] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:37.057018 19287 raft_consensus.cc:2229] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:37.057088 19287 raft_consensus.cc:2258] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:37.057410 19288 ts_tablet_manager.cc:1905] T 3d29a043334c4e55b2aea5b9f91d997a P 0ea5e748bc214b2a9f5829ab5f744ebb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:37.057542 19287 ts_tablet_manager.cc:1905] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:37.059093 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 18825
I20260620 18:39:37.059670 19030 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d29a043334c4e55b2aea5b9f91d997a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:37 UTC) from {username='oryx', principal='oryx/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:51934
I20260620 18:39:37.060985 19289 tablet_replica.cc:330] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed: stopping tablet replica
I20260620 18:39:37.061084 19289 raft_consensus.cc:2229] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:37.061146 19289 raft_consensus.cc:2258] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:37.061480 19289 ts_tablet_manager.cc:1905] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:37.063396 19289 ts_tablet_manager.cc:1918] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:37.063467 19289 log.cc:1196] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/3d29a043334c4e55b2aea5b9f91d997a
I20260620 18:39:37.063520 19287 ts_tablet_manager.cc:1918] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:37.063580 19287 log.cc:1196] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/3d29a043334c4e55b2aea5b9f91d997a
I20260620 18:39:37.063719 19289 ts_tablet_manager.cc:1939] T 3d29a043334c4e55b2aea5b9f91d997a P 9bf4bbd6fdb54e81aedd96b25f7a6fed: Deleting consensus metadata
I20260620 18:39:37.063800 19287 ts_tablet_manager.cc:1939] T 3d29a043334c4e55b2aea5b9f91d997a P 6a1beed494f744b6a28515dd3e65a324: Deleting consensus metadata
I20260620 18:39:37.064119 18739 catalog_manager.cc:4841] TS 9bf4bbd6fdb54e81aedd96b25f7a6fed (127.10.250.66:40357): tablet 3d29a043334c4e55b2aea5b9f91d997a (table test-table [id=791e1f33b2e246f78968eb868c4a4988]) successfully deleted
I20260620 18:39:37.064322 18738 catalog_manager.cc:4841] TS 6a1beed494f744b6a28515dd3e65a324 (127.10.250.67:33945): tablet 3d29a043334c4e55b2aea5b9f91d997a (table test-table [id=791e1f33b2e246f78968eb868c4a4988]) successfully deleted
W20260620 18:39:37.068804 18739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:32873 (error 108)
I20260620 18:39:37.069087 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 18965
I20260620 18:39:37.070483 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:37.069292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:32873 (local address 127.0.0.1:37230)
0620 18:39:37.070354 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:37.070422 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1039,"thread_start_us":52,"threads_started":1}
W20260620 18:39:37.070616 18739 catalog_manager.cc:4584] TS 0ea5e748bc214b2a9f5829ab5f744ebb (127.10.250.65:32873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d29a043334c4e55b2aea5b9f91d997a: Network error: Client connection negotiation failed: client connection to 127.10.250.65:32873: connect: Connection refused (error 111)
I20260620 18:39:37.075161 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 19105
I20260620 18:39:37.082037 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 18721
2026-06-20T18:39:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2522 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:37 dist-test-slave-wp6g krb5kdc[19349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:37 dist-test-slave-wp6g krb5kdc[19349](info): set up 2 sockets
Jun 20 18:39:37 dist-test-slave-wp6g krb5kdc[19349](info): commencing operation
krb5kdc: starting...
W20260620 18:39:38.123132 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:39:38 dist-test-slave-wp6g krb5kdc[19349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:39:38.245728 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38901
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:38901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:38.331454 19362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:38.331694 19362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:38.331719 19362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:38.332988 19362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:38.333043 19362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:38.333057 19362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:38.333070 19362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:38.333082 19362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:38.334738 19362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:38901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:38.335022 19362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:38.335285 19362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:38.340359 19370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:38.340391 19367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:38.340569 19368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:38.340925 19362 server_base.cc:1034] running on GCE node
I20260620 18:39:38.341144 19362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:38.341425 19362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:38.342617 19362 hybrid_clock.cc:648] HybridClock initialized: now 1781980778342587 us; error 42 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T18:39:38Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1270 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:38 dist-test-slave-wp6g krb5kdc[19375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:38 dist-test-slave-wp6g krb5kdc[19375](info): set up 2 sockets
Jun 20 18:39:38 dist-test-slave-wp6g krb5kdc[19375](info): commencing operation
krb5kdc: starting...
W20260620 18:39:39.399245 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:39:39 dist-test-slave-wp6g krb5kdc[19375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:39:39.521713 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38519
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:38519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:39.605867 19388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:39.606104 19388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:39.606149 19388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:39.607386 19388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:39.607445 19388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:39.607461 19388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:39.607476 19388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:39.607491 19388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:39.608943 19388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:38519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:39.609162 19388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:39.609351 19388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:39:39.614609 19388 server_base.cc:1034] running on GCE node
W20260620 18:39:39.614571 19394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:39.614575 19393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:39.614585 19396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:39.615172 19388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:39.615501 19388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:39.616699 19388 hybrid_clock.cc:648] HybridClock initialized: now 1781980779616698 us; error 44 us; skew 500 ppm
Jun 20 18:39:39 dist-test-slave-wp6g krb5kdc[19375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980779, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:39.622354 19388 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:39:39.622895 19388 webserver.cc:460] Webserver started at http://127.10.250.126:40257/ using document root <none> and password file <none>
I20260620 18:39:39.623142 19388 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:39.623208 19388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:39.623349 19388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:39.624485 19388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "905cef6ab071401f8a81e3cfe388bdd6"
format_stamp: "Formatted at 2026-06-20 18:39:39 on dist-test-slave-wp6g"
server_key: "34f118c30e599f6e65687334154fcc40"
server_key_iv: "e07086834423cd75414bed1880938ad0"
server_key_version: "encryptionkey@0"
I20260620 18:39:39.624858 19388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "905cef6ab071401f8a81e3cfe388bdd6"
format_stamp: "Formatted at 2026-06-20 18:39:39 on dist-test-slave-wp6g"
server_key: "34f118c30e599f6e65687334154fcc40"
server_key_iv: "e07086834423cd75414bed1880938ad0"
server_key_version: "encryptionkey@0"
I20260620 18:39:39.626231 19388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:39:39.627149 19403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:39.627346 19388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:39:39.627460 19388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "905cef6ab071401f8a81e3cfe388bdd6"
format_stamp: "Formatted at 2026-06-20 18:39:39 on dist-test-slave-wp6g"
server_key: "34f118c30e599f6e65687334154fcc40"
server_key_iv: "e07086834423cd75414bed1880938ad0"
server_key_version: "encryptionkey@0"
I20260620 18:39:39.627534 19388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:39.637926 19388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:39.640043 19388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:39.640220 19388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:39.644300 19388 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:38519
I20260620 18:39:39.644336 19455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:38519 every 8 connection(s)
I20260620 18:39:39.644733 19388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:39.645670 19456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:39.646813 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 19388
I20260620 18:39:39.646931 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:39.647120 11241 external_mini_cluster.cc:1428] Setting key 1edb32e92473b5444f42591e3f65e66a
I20260620 18:39:39.648703 19456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6: Bootstrap starting.
I20260620 18:39:39.649353 19456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:39.649662 19456 log.cc:824] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:39.650437 19456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6: No bootstrap required, opened a new log
I20260620 18:39:39.652810 19456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905cef6ab071401f8a81e3cfe388bdd6" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38519 } }
I20260620 18:39:39.652961 19456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:39.652984 19456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905cef6ab071401f8a81e3cfe388bdd6, State: Initialized, Role: FOLLOWER
I20260620 18:39:39.653118 19456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905cef6ab071401f8a81e3cfe388bdd6" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38519 } }
I20260620 18:39:39.653213 19456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:39.653237 19456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:39.653306 19456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:39.653872 19456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905cef6ab071401f8a81e3cfe388bdd6" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38519 } }
I20260620 18:39:39.654003 19456 leader_election.cc:302] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 905cef6ab071401f8a81e3cfe388bdd6; no voters:
I20260620 18:39:39.654203 19456 leader_election.cc:288] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:39.654233 19461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:39.654349 19461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 1 LEADER]: Becoming Leader. State: Replica: 905cef6ab071401f8a81e3cfe388bdd6, State: Running, Role: LEADER
Jun 20 18:39:39 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:39.654438 19461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905cef6ab071401f8a81e3cfe388bdd6" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38519 } }
I20260620 18:39:39.654489 19456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:39.655611 19463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 905cef6ab071401f8a81e3cfe388bdd6. Latest consensus state: current_term: 1 leader_uuid: "905cef6ab071401f8a81e3cfe388bdd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905cef6ab071401f8a81e3cfe388bdd6" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38519 } } }
I20260620 18:39:39.655714 19463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:39.656061 19462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "905cef6ab071401f8a81e3cfe388bdd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905cef6ab071401f8a81e3cfe388bdd6" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38519 } } }
I20260620 18:39:39.656402 19462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:39.655905 19475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:39.657012 19475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:39.658728 19475 catalog_manager.cc:1347] Generated new cluster ID: 79a8b72b1dca4efb9750c929cedf8d55
I20260620 18:39:39.658767 19475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:39.658938 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:39.647672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.10.250.126:38519)
0620 18:39:39.647881 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:39.647892 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:39.647928 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:39:39.648455 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:39.648465 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:39.648527 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:39.648684 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:39.649323 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:39.650696 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:39.651460 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:39.652108 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:39.652250 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:39.655026 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:39.655030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:39.655031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:39.655045 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:39.657207 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:39.657652 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:39.657655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:39.657656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:39.657758 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:39.658000 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:39.658001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:39.658001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:39.658248 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:39.658410 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:39.658552 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 18:39:39.658703 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":95,"threads_started":1}
I20260620 18:39:39.671618 19475 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:39.672458 19475 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:39.678658 19475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6: Generated new TSK 0
I20260620 18:39:39.678915 19475 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:39:39.710250 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38519
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:39.797191 19482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:39.797446 19482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:39.797487 19482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:39.798730 19482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:39.798796 19482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:39.798841 19482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:39.800338 19482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:39.800621 19482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:39.800854 19482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:39.805897 19487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:39.805959 19482 server_base.cc:1034] running on GCE node
W20260620 18:39:39.805927 19490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:39.805929 19488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:39.806372 19482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:39.806691 19482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:39.807884 19482 hybrid_clock.cc:648] HybridClock initialized: now 1781980779807873 us; error 33 us; skew 500 ppm
Jun 20 18:39:39 dist-test-slave-wp6g krb5kdc[19375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980779, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:39.810696 19482 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:39:39.811210 19482 webserver.cc:460] Webserver started at http://127.10.250.65:42683/ using document root <none> and password file <none>
I20260620 18:39:39.811470 19482 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:39.811539 19482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:39.811690 19482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:39.812803 19482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "22fc9d458d0d4806b5540a1313138ece"
format_stamp: "Formatted at 2026-06-20 18:39:39 on dist-test-slave-wp6g"
server_key: "b50ef092cb4228789703a6ef725f0b40"
server_key_iv: "ab76a4822993d0b087c1aa8c78163759"
server_key_version: "encryptionkey@0"
I20260620 18:39:39.813301 19482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "22fc9d458d0d4806b5540a1313138ece"
format_stamp: "Formatted at 2026-06-20 18:39:39 on dist-test-slave-wp6g"
server_key: "b50ef092cb4228789703a6ef725f0b40"
server_key_iv: "ab76a4822993d0b087c1aa8c78163759"
server_key_version: "encryptionkey@0"
I20260620 18:39:39.814875 19482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:39.815841 19497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:39.816072 19482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:39.816179 19482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "22fc9d458d0d4806b5540a1313138ece"
format_stamp: "Formatted at 2026-06-20 18:39:39 on dist-test-slave-wp6g"
server_key: "b50ef092cb4228789703a6ef725f0b40"
server_key_iv: "ab76a4822993d0b087c1aa8c78163759"
server_key_version: "encryptionkey@0"
I20260620 18:39:39.816257 19482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:39.828457 19482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:39.831122 19482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:39.831288 19482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:39.831539 19482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:39.831887 19482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:39.831948 19482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:39.831995 19482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:39.832024 19482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:39.838068 19482 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:45129
I20260620 18:39:39.838112 19610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:45129 every 8 connection(s)
I20260620 18:39:39.838459 19482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:39:39 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980779, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:39.845651 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 19482
I20260620 18:39:39.845744 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:39.845953 11241 external_mini_cluster.cc:1428] Setting key 9f24dab8e1680252bd298cc55875216a
I20260620 18:39:39.848268 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:39.839247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:41411 (local address 127.10.250.126:38519)
0620 18:39:39.839316 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:39.839318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:39.839508 (+ 190us) server_negotiation.cc:407] Connection header received
0620 18:39:39.839989 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:39.839990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:39.840007 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:39.840068 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:39.841089 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:39.841610 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:39.842738 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:39.842960 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:39.843031 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:39.845133 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:39.845135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:39.845135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:39.845143 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:39.846825 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:39.847248 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:39.847250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:39.847250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:39.847313 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:39.847597 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:39.847599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:39.847599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:39.847721 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:39.847790 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:39.848079 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 18:39:39.848172 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:39:39.848500 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:39.839202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:41411)
0620 18:39:39.839367 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:39.839420 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:39.839905 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:39.840125 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:39.840128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:39.840756 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:39.840966 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:39.840971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:39.841709 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:39.841710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:39.842637 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:39.842640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:39.843047 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:39.843048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:39.843153 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:39.843654 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:39.843662 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:39.845027 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:39.846924 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:39.846932 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:39.846935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:39.847171 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:39.847422 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:39.847423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:39.847423 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:39.847531 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:39.847792 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:39.847800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:39.847944 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 18:39:39.848057 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:39:39.848231 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260620 18:39:39.848959 19611 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38519
I20260620 18:39:39.849080 19611 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:39.849264 19611 heartbeater.cc:511] Master 127.10.250.126:38519 requested a full tablet report, sending...
I20260620 18:39:39.849812 19420 ts_manager.cc:194] Registered new tserver with Master: 22fc9d458d0d4806b5540a1313138ece (127.10.250.65:45129)
I20260620 18:39:39.850440 19420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:41411
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:39:39.872887 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38519
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:39.958616 19616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:39.958880 19616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:39.958910 19616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:39.960232 19616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:39.960289 19616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:39.960319 19616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:39.961905 19616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:39.962163 19616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:39.962405 19616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:39.967760 19621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:39.967736 19622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:39.967729 19624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:39.967844 19616 server_base.cc:1034] running on GCE node
I20260620 18:39:39.968201 19616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:39.968482 19616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:39.969642 19616 hybrid_clock.cc:648] HybridClock initialized: now 1781980779969621 us; error 36 us; skew 500 ppm
Jun 20 18:39:39 dist-test-slave-wp6g krb5kdc[19375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980779, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:39.972394 19616 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:39:39.972841 19616 webserver.cc:460] Webserver started at http://127.10.250.66:38867/ using document root <none> and password file <none>
I20260620 18:39:39.973101 19616 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:39.973151 19616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:39.973281 19616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:39.974442 19616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "6fcd2fd2281f4daf9b72bfab3e72232c"
format_stamp: "Formatted at 2026-06-20 18:39:39 on dist-test-slave-wp6g"
server_key: "82742c36bf64b134e37285975458a731"
server_key_iv: "a6dde6f3c6083bb14e4de7ee88d14a77"
server_key_version: "encryptionkey@0"
I20260620 18:39:39.974886 19616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "6fcd2fd2281f4daf9b72bfab3e72232c"
format_stamp: "Formatted at 2026-06-20 18:39:39 on dist-test-slave-wp6g"
server_key: "82742c36bf64b134e37285975458a731"
server_key_iv: "a6dde6f3c6083bb14e4de7ee88d14a77"
server_key_version: "encryptionkey@0"
I20260620 18:39:39.976331 19616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:39:39.977239 19631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:39.977593 19616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:39.977679 19616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "6fcd2fd2281f4daf9b72bfab3e72232c"
format_stamp: "Formatted at 2026-06-20 18:39:39 on dist-test-slave-wp6g"
server_key: "82742c36bf64b134e37285975458a731"
server_key_iv: "a6dde6f3c6083bb14e4de7ee88d14a77"
server_key_version: "encryptionkey@0"
I20260620 18:39:39.977805 19616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:39.998687 19616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:40.001228 19616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:40.001360 19616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:40.001681 19616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:40.002050 19616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:40.002105 19616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:40.002151 19616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:40.002173 19616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:40.009846 19616 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:40199
I20260620 18:39:40.009938 19744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:40199 every 8 connection(s)
I20260620 18:39:40.010370 19616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980779, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:40.018497 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 19616
I20260620 18:39:40.018680 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:40.018934 11241 external_mini_cluster.cc:1428] Setting key a85e061c954e9b1ec958afbd7e728d1b
I20260620 18:39:40.021440 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:40.011218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:50711 (local address 127.10.250.126:38519)
0620 18:39:40.011278 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:40.011280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:40.011540 (+ 260us) server_negotiation.cc:407] Connection header received
0620 18:39:40.012117 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:40.012119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:40.012132 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:40.012178 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:40.013207 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.013736 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.014890 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.015110 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.015198 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.017647 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:40.017649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:40.017649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:40.017658 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:40.019477 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:40.020086 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:40.020088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:40.020088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:40.020161 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:40.020548 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:40.020549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:40.020550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:40.020792 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:40.020878 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:40.021184 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 18:39:40.021304 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:39:40.021780 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:40.011219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:50711)
0620 18:39:40.011393 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:40.011433 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:40.011994 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:40.012272 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:40.012282 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:40.012885 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:40.013102 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.013108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.013849 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.013850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.014803 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.014806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.015210 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.015212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.015346 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.015919 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:40.015928 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:40.017500 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:40.019557 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:40.019566 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:40.019568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:40.020002 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:40.020285 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:40.020285 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:40.020286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:40.020475 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:40.020889 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:40.020896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:40.021042 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 18:39:40.021144 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:39:40.021290 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260620 18:39:40.022430 19745 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38519
I20260620 18:39:40.022573 19745 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:40.022830 19745 heartbeater.cc:511] Master 127.10.250.126:38519 requested a full tablet report, sending...
I20260620 18:39:40.023240 19420 ts_manager.cc:194] Registered new tserver with Master: 6fcd2fd2281f4daf9b72bfab3e72232c (127.10.250.66:40199)
I20260620 18:39:40.023694 19420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:50711
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:39:40.047966 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38519
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:40.131639 19750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:40.131886 19750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:40.131910 19750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:40.133167 19750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:40.133213 19750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:40.133241 19750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:40.134778 19750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:40.135043 19750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:40.135279 19750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:40.140337 19756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:40.140328 19755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:40.140342 19758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:40.140473 19750 server_base.cc:1034] running on GCE node
I20260620 18:39:40.140795 19750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:40.141031 19750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:40.142206 19750 hybrid_clock.cc:648] HybridClock initialized: now 1781980780142192 us; error 31 us; skew 500 ppm
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980780, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:40.144888 19750 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:39:40.145431 19750 webserver.cc:460] Webserver started at http://127.10.250.67:42121/ using document root <none> and password file <none>
I20260620 18:39:40.145668 19750 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:40.145733 19750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:40.145887 19750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:40.147141 19750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "ae1c242222954b61832299cb55af1185"
format_stamp: "Formatted at 2026-06-20 18:39:40 on dist-test-slave-wp6g"
server_key: "79277035ccb8a3672da5a10df57d3712"
server_key_iv: "fd86b6ce07f79326d31a33125b9ed7b6"
server_key_version: "encryptionkey@0"
I20260620 18:39:40.147535 19750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "ae1c242222954b61832299cb55af1185"
format_stamp: "Formatted at 2026-06-20 18:39:40 on dist-test-slave-wp6g"
server_key: "79277035ccb8a3672da5a10df57d3712"
server_key_iv: "fd86b6ce07f79326d31a33125b9ed7b6"
server_key_version: "encryptionkey@0"
I20260620 18:39:40.148955 19750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:39:40.149852 19765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:40.150076 19750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:40.150249 19750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "ae1c242222954b61832299cb55af1185"
format_stamp: "Formatted at 2026-06-20 18:39:40 on dist-test-slave-wp6g"
server_key: "79277035ccb8a3672da5a10df57d3712"
server_key_iv: "fd86b6ce07f79326d31a33125b9ed7b6"
server_key_version: "encryptionkey@0"
I20260620 18:39:40.150336 19750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:40.167344 19750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:40.169638 19750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:40.169828 19750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:40.170080 19750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:40.170424 19750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:40.170486 19750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:40.170522 19750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:40.170600 19750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:40.177139 19750 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:40289
I20260620 18:39:40.177182 19878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:40289 every 8 connection(s)
I20260620 18:39:40.177536 19750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:40.183568 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 19750
I20260620 18:39:40.183701 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:40.183923 11241 external_mini_cluster.cc:1428] Setting key 530d5a1fe692894d078f8b27df571d38
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980780, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:40.188761 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:40.178299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:40921 (local address 127.10.250.126:38519)
0620 18:39:40.178387 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:40.178390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:40.178553 (+ 163us) server_negotiation.cc:407] Connection header received
0620 18:39:40.179235 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:40.179237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:40.179251 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:40.179345 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:40.180555 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.181461 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.182989 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.183190 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.183258 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.185725 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:40.185727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:40.185727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:40.185734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:40.187215 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:40.187645 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:40.187646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:40.187647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:40.187706 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:40.188050 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:40.188051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:40.188051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:40.188183 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:40.188249 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:40.188598 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 18:39:40.188688 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:39:40.188979 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:40.178245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:40921)
0620 18:39:40.178399 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:40.178448 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:40.179101 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:40.179414 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:40.179418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:40.180059 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:40.180371 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.180378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.181585 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.181586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.182889 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.182892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.183270 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.183271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.183375 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.183989 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:40.184016 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:40.185640 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:40.187287 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:40.187293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:40.187295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:40.187534 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:40.187817 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:40.187818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:40.187818 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:40.187951 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:40.188268 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:40.188274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:40.188434 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 18:39:40.188543 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:39:40.188690 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260620 18:39:40.189745 19879 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38519
I20260620 18:39:40.189857 19879 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:40.190068 19879 heartbeater.cc:511] Master 127.10.250.126:38519 requested a full tablet report, sending...
I20260620 18:39:40.190606 19420 ts_manager.cc:194] Registered new tserver with Master: ae1c242222954b61832299cb55af1185 (127.10.250.67:40289)
I20260620 18:39:40.190996 19420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:40921
I20260620 18:39:40.195775 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.65.keytab.
Entry for principal oryx/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.65.keytab.
WARNING: no policy specified for oryx/127.10.250.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.66.keytab.
Entry for principal oryx/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.66.keytab.
WARNING: no policy specified for oryx/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.67.keytab.
Entry for principal oryx/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.67.keytab.
I20260620 18:39:40.273510 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 19482
I20260620 18:39:40.277998 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 19616
I20260620 18:39:40.283545 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 19750
I20260620 18:39:40.287563 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 19388
I20260620 18:39:40.292762 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38519
--webserver_interface=127.10.250.126
--webserver_port=40257
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:38519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:40.375520 19888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:40.375764 19888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:40.375789 19888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:40.377032 19888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:40.377084 19888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:40.377099 19888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:40.377111 19888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:40.377123 19888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:40.378821 19888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:38519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=40257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:40.379137 19888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:40.379391 19888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:40.384775 19896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:40.384775 19893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:40.384812 19894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:40.385021 19888 server_base.cc:1034] running on GCE node
I20260620 18:39:40.385376 19888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:40.385722 19888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:40.386910 19888 hybrid_clock.cc:648] HybridClock initialized: now 1781980780386893 us; error 35 us; skew 500 ppm
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980780, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:40.389720 19888 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:39:40.390216 19888 webserver.cc:460] Webserver started at http://127.10.250.126:40257/ using document root <none> and password file <none>
I20260620 18:39:40.390496 19888 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:40.390625 19888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:40.392431 19888 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:39:40.393225 19903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:40.393465 19888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:40.393545 19888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "905cef6ab071401f8a81e3cfe388bdd6"
format_stamp: "Formatted at 2026-06-20 18:39:39 on dist-test-slave-wp6g"
server_key: "34f118c30e599f6e65687334154fcc40"
server_key_iv: "e07086834423cd75414bed1880938ad0"
server_key_version: "encryptionkey@0"
I20260620 18:39:40.393901 19888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:40.411985 19888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:40.414551 19888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:40.414680 19888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:40.418623 19888 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:38519
I20260620 18:39:40.418727 19955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:38519 every 8 connection(s)
I20260620 18:39:40.419106 19888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:40.420658 19956 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:39:40.421424 19956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6: Bootstrap starting.
I20260620 18:39:40.428021 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 19888
I20260620 18:39:40.428406 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:45129
--local_ip_for_outbound_sockets=127.10.250.65
--tserver_master_addrs=127.10.250.126:38519
--webserver_port=42683
--webserver_interface=127.10.250.65
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.250.65
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:39:40.430823 19956 log.cc:824] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:40.432672 19956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:39:40.432998 19956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6: Bootstrap complete.
I20260620 18:39:40.435439 19956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905cef6ab071401f8a81e3cfe388bdd6" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38519 } }
I20260620 18:39:40.435709 19956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905cef6ab071401f8a81e3cfe388bdd6, State: Initialized, Role: FOLLOWER
I20260620 18:39:40.435925 19956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905cef6ab071401f8a81e3cfe388bdd6" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38519 } }
I20260620 18:39:40.436020 19956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:40.436074 19956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:40.436124 19956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:39:40.437218 19956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905cef6ab071401f8a81e3cfe388bdd6" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38519 } }
I20260620 18:39:40.437348 19956 leader_election.cc:302] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 905cef6ab071401f8a81e3cfe388bdd6; no voters:
I20260620 18:39:40.437566 19956 leader_election.cc:288] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:39:40.437609 19960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:39:40.437731 19960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [term 2 LEADER]: Becoming Leader. State: Replica: 905cef6ab071401f8a81e3cfe388bdd6, State: Running, Role: LEADER
I20260620 18:39:40.437866 19956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:40.437901 19960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905cef6ab071401f8a81e3cfe388bdd6" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38519 } }
I20260620 18:39:40.438398 19961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 905cef6ab071401f8a81e3cfe388bdd6. Latest consensus state: current_term: 2 leader_uuid: "905cef6ab071401f8a81e3cfe388bdd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905cef6ab071401f8a81e3cfe388bdd6" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38519 } } }
I20260620 18:39:40.438681 19961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:40.438416 19962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "905cef6ab071401f8a81e3cfe388bdd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905cef6ab071401f8a81e3cfe388bdd6" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38519 } } }
I20260620 18:39:40.438982 19962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:40.439911 19975 catalog_manager.cc:1259] Loaded cluster ID: 79a8b72b1dca4efb9750c929cedf8d55
I20260620 18:39:40.439971 19975 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6: loading cluster ID for follower catalog manager: success
I20260620 18:39:40.440871 19975 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6: acquiring CA information for follower catalog manager: success
I20260620 18:39:40.441185 19975 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:39:40.441313 19969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:40.441484 19969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:40.441535 19969 catalog_manager.cc:1259] Loaded cluster ID: 79a8b72b1dca4efb9750c929cedf8d55
I20260620 18:39:40.441550 19969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:40.441656 19969 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:40.441704 19969 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 905cef6ab071401f8a81e3cfe388bdd6: Loaded TSK: 0
I20260620 18:39:40.441790 19969 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:39:40.529605 19958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:40.529865 19958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:40.529906 19958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:40.531224 19958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:40.531299 19958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:40.531347 19958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:40.532908 19958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.65.keytab
--principal=oryx/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:45129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=42683
--tserver_master_addrs=127.10.250.126:38519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:40.533205 19958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:40.533458 19958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:40.539628 19983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:40.539666 19982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:40.539610 19985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:40.540104 19958 server_base.cc:1034] running on GCE node
I20260620 18:39:40.540328 19958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:40.540608 19958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:40.541851 19958 hybrid_clock.cc:648] HybridClock initialized: now 1781980780541835 us; error 32 us; skew 500 ppm
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980780, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:40.544876 19958 init.cc:377] Logged in from keytab as oryx/127.10.250.65@KRBTEST.COM (short username oryx)
I20260620 18:39:40.545450 19958 webserver.cc:460] Webserver started at http://127.10.250.65:42683/ using document root <none> and password file <none>
I20260620 18:39:40.545746 19958 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:40.545836 19958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:40.547748 19958 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:40.548570 19992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:40.548794 19958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:40.548905 19958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "22fc9d458d0d4806b5540a1313138ece"
format_stamp: "Formatted at 2026-06-20 18:39:39 on dist-test-slave-wp6g"
server_key: "b50ef092cb4228789703a6ef725f0b40"
server_key_iv: "ab76a4822993d0b087c1aa8c78163759"
server_key_version: "encryptionkey@0"
I20260620 18:39:40.549369 19958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:40.571133 19958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:40.574100 19958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:40.574267 19958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:40.574584 19958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:40.574955 19958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:40.574996 19958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:40.575029 19958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:40.575047 19958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:40.582823 19958 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:45129
I20260620 18:39:40.582947 20105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:45129 every 8 connection(s)
I20260620 18:39:40.583348 19958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:40.584766 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 19958
I20260620 18:39:40.585095 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:40199
--local_ip_for_outbound_sockets=127.10.250.66
--tserver_master_addrs=127.10.250.126:38519
--webserver_port=38867
--webserver_interface=127.10.250.66
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.250.66
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:40.596017 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:40.584166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:43399)
0620 18:39:40.584382 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:40.584427 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:40.585293 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:40.587271 (+ 1978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:40.587276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:40.587926 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:40.588141 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.588147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.589900 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.589901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.590948 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.590951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.591815 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.591816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.591984 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.592688 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:40.592698 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:40.595790 (+ 3092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":138,"threads_started":1}
W20260620 18:39:40.596297 20106 heartbeater.cc:644] Failed to heartbeat to 127.10.250.126:38519 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.250.126:38519: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
W20260620 18:39:40.596688 20109 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:40.584246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:43399 (local address 127.10.250.126:38519)
0620 18:39:40.586706 (+ 2460us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:40.586717 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:40.586770 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:39:40.586938 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:40.586946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:40.587010 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:40.587191 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:40.588291 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.589744 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.591051 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.591688 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.591812 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.596025 (+ 4213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:43399: BlockingRecv error: recv got EOF from 127.10.250.65:43399 (error 108)
Metrics: {"server-negotiator.queue_time_us":2382,"thread_start_us":101,"threads_started":1}
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:40.602988 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:40.596535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:56141)
0620 18:39:40.596912 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:40.596924 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:40.596975 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:40.597268 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:40.597269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:40.597445 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:40.597514 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.597516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.598587 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.598588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.599434 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.599437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.599949 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.599950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.600032 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.600579 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:40.600585 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:40.602904 (+ 2319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":101,"threads_started":1}
W20260620 18:39:40.603083 20109 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:40.596732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:56141 (local address 127.10.250.126:38519)
0620 18:39:40.597111 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:40.597114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:40.597126 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:39:40.597156 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:40.597157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:40.597169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:40.597242 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:40.597616 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.598456 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.599541 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.599867 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.599940 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.603016 (+ 3076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:56141: BlockingRecv error: recv got EOF from 127.10.250.65:56141 (error 108)
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":95,"threads_started":1}
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:40.608870 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:40.603540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:54805)
0620 18:39:40.603586 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:40.603596 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:40.603677 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:40.603835 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:40.603836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:40.603984 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:40.604026 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.604027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.604932 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.604933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.605746 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.605747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.606224 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.606225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.606278 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.606788 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:40.606794 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:40.608799 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 18:39:40.609035 20106 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:39:40.609263 20109 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:40.603605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:54805 (local address 127.10.250.126:38519)
0620 18:39:40.603644 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:40.603646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:40.603654 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:39:40.603746 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:40.603747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:40.603759 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:40.603822 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:40.604092 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.604843 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.605832 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.606147 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.606218 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.608900 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:54805: BlockingRecv error: recv got EOF from 127.10.250.65:54805 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 18:39:40.675220 20110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:40.675448 20110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:40.675475 20110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:40.676719 20110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:40.676765 20110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:40.676795 20110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:40.678301 20110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.66.keytab
--principal=oryx/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:40199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=38867
--tserver_master_addrs=127.10.250.126:38519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:40.678582 20110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:40.678850 20110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:40.684309 20117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:40.684422 20110 server_base.cc:1034] running on GCE node
W20260620 18:39:40.684315 20118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:40.684315 20120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:40.684793 20110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:40.685027 20110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:40.686183 20110 hybrid_clock.cc:648] HybridClock initialized: now 1781980780686167 us; error 29 us; skew 500 ppm
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980780, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:40.688768 20110 init.cc:377] Logged in from keytab as oryx/127.10.250.66@KRBTEST.COM (short username oryx)
I20260620 18:39:40.689194 20110 webserver.cc:460] Webserver started at http://127.10.250.66:38867/ using document root <none> and password file <none>
I20260620 18:39:40.689404 20110 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:40.689458 20110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:40.690865 20110 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:40.691588 20127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:40.691789 20110 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:39:40.691880 20110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "6fcd2fd2281f4daf9b72bfab3e72232c"
format_stamp: "Formatted at 2026-06-20 18:39:39 on dist-test-slave-wp6g"
server_key: "82742c36bf64b134e37285975458a731"
server_key_iv: "a6dde6f3c6083bb14e4de7ee88d14a77"
server_key_version: "encryptionkey@0"
I20260620 18:39:40.692216 20110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:40.721014 20110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:40.723543 20110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:40.723668 20110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:40.723946 20110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:40.724306 20110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:40.724349 20110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:40.724375 20110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:40.724387 20110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:40.730101 20110 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:40199
I20260620 18:39:40.730188 20240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:40199 every 8 connection(s)
I20260620 18:39:40.730582 20110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:40.734338 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 20110
I20260620 18:39:40.734726 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:40289
--local_ip_for_outbound_sockets=127.10.250.67
--tserver_master_addrs=127.10.250.126:38519
--webserver_port=42121
--webserver_interface=127.10.250.67
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.250.67
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:40.739603 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:40.731363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:55983)
0620 18:39:40.731528 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:40.731575 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:40.732507 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:40.732774 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:40.732778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:40.733043 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:40.733243 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.733247 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.733990 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.733991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.735162 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.735165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.735729 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.735730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.735925 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.736599 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:40.736612 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:40.739441 (+ 2829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
W20260620 18:39:40.739665 20109 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:40.731418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:55983 (local address 127.10.250.126:38519)
0620 18:39:40.731465 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:40.731467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:40.731691 (+ 224us) server_negotiation.cc:407] Connection header received
0620 18:39:40.732616 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:40.732617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:40.732632 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:40.732697 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:40.733373 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.733873 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.735290 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.735618 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.735703 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.739575 (+ 3872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:55983: BlockingRecv error: recv got EOF from 127.10.250.66:55983 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 18:39:40.739882 20241 heartbeater.cc:644] Failed to heartbeat to 127.10.250.126:38519 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.250.126:38519: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:40.744583 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:40.740136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:44197)
0620 18:39:40.740180 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:40.740190 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:40.740230 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:40.740355 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:40.740356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:40.740494 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:40.740540 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.740541 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.741175 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.741176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.741693 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.741694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.742057 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.742058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.742124 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.742672 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:40.742678 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:40.744532 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 18:39:40.744736 20109 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:40.740189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:44197 (local address 127.10.250.126:38519)
0620 18:39:40.740247 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:40.740249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:40.740257 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:39:40.740286 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:40.740286 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:40.740303 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:40.740354 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:40.740621 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.741099 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.741773 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.741981 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.742049 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.744652 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:44197: BlockingRecv error: recv got EOF from 127.10.250.66:44197 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:40.749756 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:40.744960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:34773)
0620 18:39:40.745020 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:40.745033 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:40.745084 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:40.745255 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:40.745256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:40.745403 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:40.745441 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.745443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.746065 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.746066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.746918 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.746920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.747261 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.747261 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.747314 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.747736 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:40.747741 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:40.749682 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 18:39:40.749862 20109 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:40.745013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:34773 (local address 127.10.250.126:38519)
0620 18:39:40.745056 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:40.745059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:40.745068 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:40.745144 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:40.745145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:40.745154 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:40.745208 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:40.745522 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.746003 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.747004 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.747190 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.747252 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.749777 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:34773: BlockingRecv error: recv got EOF from 127.10.250.66:34773 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 18:39:40.749894 20241 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:39:40.819155 20244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:40.819371 20244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:40.819406 20244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:40.820559 20244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:40.820597 20244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:40.820624 20244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:40.822014 20244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/krb5kdc/oryx_127.10.250.67.keytab
--principal=oryx/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:40289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=42121
--tserver_master_addrs=127.10.250.126:38519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:40.822212 20244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:40.822391 20244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:40.827574 20252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:40.827607 20250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:40.827607 20249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:40.827704 20244 server_base.cc:1034] running on GCE node
I20260620 18:39:40.828117 20244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:40.828406 20244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:40.829576 20244 hybrid_clock.cc:648] HybridClock initialized: now 1781980780829561 us; error 42 us; skew 500 ppm
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980780, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:40.832181 20244 init.cc:377] Logged in from keytab as oryx/127.10.250.67@KRBTEST.COM (short username oryx)
I20260620 18:39:40.832645 20244 webserver.cc:460] Webserver started at http://127.10.250.67:42121/ using document root <none> and password file <none>
I20260620 18:39:40.832894 20244 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:40.832978 20244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:40.834456 20244 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:39:40.835215 20259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:40.835445 20244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:40.835552 20244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "ae1c242222954b61832299cb55af1185"
format_stamp: "Formatted at 2026-06-20 18:39:40 on dist-test-slave-wp6g"
server_key: "79277035ccb8a3672da5a10df57d3712"
server_key_iv: "fd86b6ce07f79326d31a33125b9ed7b6"
server_key_version: "encryptionkey@0"
I20260620 18:39:40.835943 20244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:40.853821 20244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:40.856745 20244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:40.856915 20244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:40.857185 20244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:40.857504 20244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:40.857553 20244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:40.857595 20244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:40.857622 20244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:40.863801 20244 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:40289
I20260620 18:39:40.863884 20372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:40289 every 8 connection(s)
I20260620 18:39:40.864252 20244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:40.870599 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 20244
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:39:40.872112 20109 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:40.865088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:60805 (local address 127.10.250.126:38519)
0620 18:39:40.865177 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:40.865178 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:40.865344 (+ 166us) server_negotiation.cc:407] Connection header received
0620 18:39:40.865840 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:40.865842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:40.865853 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:40.865926 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:40.866982 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.867481 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.868496 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.868681 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.868738 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.872064 (+ 3326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:60805: BlockingRecv error: recv got EOF from 127.10.250.67:60805 (error 108)
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":48}
I20260620 18:39:40.872186 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:40.865053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:60805)
0620 18:39:40.865216 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:40.865259 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:40.865742 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:40.865999 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:40.866003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:40.866667 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:40.866886 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.866892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.867586 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.867587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.868415 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.868418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.868775 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.868776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.868902 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.869432 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:40.869441 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:40.871972 (+ 2531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
W20260620 18:39:40.872458 20373 heartbeater.cc:644] Failed to heartbeat to 127.10.250.126:38519 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.250.126:38519: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:40.877398 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:40.872666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:56333)
0620 18:39:40.872715 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:40.872726 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:40.872776 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:40.872903 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:40.872903 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:40.873032 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:40.873075 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.873076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.873699 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.873700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.874276 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.874278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.874695 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.874696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.874768 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.875180 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:40.875186 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:40.877292 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 18:39:40.877424 20109 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:40.872716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:56333 (local address 127.10.250.126:38519)
0620 18:39:40.872758 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:40.872760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:40.872770 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:39:40.872835 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:40.872836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:40.872846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:40.872893 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:40.873153 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.873609 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.874366 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.874617 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.874694 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.877371 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:56333: BlockingRecv error: recv got EOF from 127.10.250.67:56333 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:39:40.879068 20112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:40.871154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.10.250.126:38519)
0620 18:39:40.871231 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:40.871233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:40.871241 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:39:40.871265 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:40.871266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:40.871278 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:40.871344 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:40.871958 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.872634 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.873451 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.873655 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.873729 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.875134 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:40.875140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:40.875141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:40.875156 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:40.877418 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:40.877845 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:40.877849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:40.877849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:40.877998 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:40.878195 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:40.878195 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:40.878196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:40.878493 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:40.878661 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:40.878796 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 18:39:40.878932 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:40 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:40.882166 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:40.877807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:56303)
0620 18:39:40.877851 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:40.877862 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:40.877906 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:40.878010 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:40.878011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:40.878150 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:40.878204 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.878206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.879016 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.879017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.879652 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:40.879655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.880120 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.880121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:40.880161 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.880584 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:40.880589 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:40.882118 (+ 1529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 18:39:40.882251 20109 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:40.877847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:56303 (local address 127.10.250.126:38519)
0620 18:39:40.877923 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:40.877925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:40.877931 (+ 6us) server_negotiation.cc:407] Connection header received
0620 18:39:40.877949 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:40.877950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:40.877957 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:40.878004 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:40.878274 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.878927 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.879777 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:40.880064 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:40.880113 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:40.882190 (+ 2077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:56303: BlockingRecv error: recv got EOF from 127.10.250.67:56303 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 18:39:40.882349 20373 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 18:39:41 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:41 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:41.614810 20377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:41.609493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:40197)
0620 18:39:41.609617 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:41.609630 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:41.609712 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:41.609887 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:41.609888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:41.610049 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:41.610099 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:41.610101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:41.610891 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:41.610892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:41.611635 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:41.611638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:41.612040 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:41.612041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:41.612084 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:41.612577 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:41.612582 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:41.614697 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
W20260620 18:39:41.614851 20378 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:41.609586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:40197 (local address 127.10.250.126:38519)
0620 18:39:41.609739 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:41.609741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:41.609751 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:39:41.609776 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:41.609777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:41.609787 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:41.609849 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:41.610198 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:41.610782 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:41.611740 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:41.611953 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:41.612021 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:41.614776 (+ 2755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:40197: BlockingRecv error: recv got EOF from 127.10.250.65:40197 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
Jun 20 18:39:41 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:41 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:41.755628 20379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:41.750329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:34843)
0620 18:39:41.750502 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:41.750517 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:41.750631 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:41.750820 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:41.750821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:41.751030 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:41.751083 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:41.751085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:41.751834 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:41.751836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:41.752400 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:41.752402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:41.752828 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:41.752829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:41.752887 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:41.753424 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:41.753435 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:41.755517 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
W20260620 18:39:41.755687 20378 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:41.750452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:34843 (local address 127.10.250.126:38519)
0620 18:39:41.750573 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:41.750575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:41.750606 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:39:41.750716 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:41.750716 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:41.750737 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:41.750813 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:41.751193 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:41.751734 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:41.752512 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:41.752733 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:41.752828 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:41.755619 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:34843: BlockingRecv error: recv got EOF from 127.10.250.66:34843 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 18:39:41 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:41 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:41.888973 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:41.882796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:39743)
0620 18:39:41.883077 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:41.883090 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:41.883191 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:41.883388 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:41.883389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:41.883577 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:41.883640 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:41.883642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:41.884332 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:41.884333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:41.884898 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:41.884899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:41.885363 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:41.885364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:41.885471 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:41.886276 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:41.886282 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:41.888828 (+ 2546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":181,"threads_started":1}
W20260620 18:39:41.889262 20378 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:41.883003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:39743 (local address 127.10.250.126:38519)
0620 18:39:41.883097 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:41.883098 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:41.883174 (+ 76us) server_negotiation.cc:407] Connection header received
0620 18:39:41.883314 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:41.883315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:41.883329 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:41.883383 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:41.883745 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:41.884249 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:41.884989 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:41.885220 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:41.885306 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:41.888966 (+ 3660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:39743: BlockingRecv error: recv got EOF from 127.10.250.67:39743 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 18:39:42 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:42 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:42.620631 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.615506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:53711)
0620 18:39:42.615704 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:42.615715 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:42.615791 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:42.616021 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:42.616022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:42.616246 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:42.616309 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.616312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.617080 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.617081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.617640 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.617642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.618030 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.618031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.618075 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.618773 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:42.618778 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:42.620572 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":93,"threads_started":1}
W20260620 18:39:42.620844 20382 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:42.615648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:53711 (local address 127.10.250.126:38519)
0620 18:39:42.615790 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:42.615792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:42.615804 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:39:42.615884 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:42.615885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:42.615898 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:42.616001 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:42.616397 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.616927 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.617734 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.617954 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.618021 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.620732 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:53711: BlockingRecv error: recv got EOF from 127.10.250.65:53711 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
Jun 20 18:39:42 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:42 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:42.761384 20383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.756251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:38695)
0620 18:39:42.756387 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:42.756398 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:42.756451 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:42.756594 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:42.756595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:42.756773 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:42.756823 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.756825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.757548 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.757550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.758401 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.758405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.758887 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.758888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.758940 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.759483 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:42.759489 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:42.761306 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
W20260620 18:39:42.761587 20382 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:42.756349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:38695 (local address 127.10.250.126:38519)
0620 18:39:42.756435 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:42.756437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:42.756448 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:39:42.756520 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:42.756521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:42.756532 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:42.756576 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:42.756927 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.757440 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.758595 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.758799 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.758857 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.761512 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:38695: BlockingRecv error: recv got EOF from 127.10.250.66:38695 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 18:39:42 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:42 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:42.894318 20384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.889520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:41047)
0620 18:39:42.889635 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:42.889648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:42.889711 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:42.889869 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:42.889871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:42.890041 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:42.890090 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.890091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.890839 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.890840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.891366 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.891368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.891704 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.891705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.891766 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.892294 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:42.892300 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:42.894240 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":62,"threads_started":1}
W20260620 18:39:42.894426 20382 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:42.889612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:41047 (local address 127.10.250.126:38519)
0620 18:39:42.889659 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:42.889661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:42.889694 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:39:42.889760 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:42.889761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:42.889772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:42.889821 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:42.890166 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.890758 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.891439 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.891618 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.891673 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.894327 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:41047: BlockingRecv error: recv got EOF from 127.10.250.67:41047 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 18:39:43 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:43 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:43.626490 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:43.621182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:60753)
0620 18:39:43.621388 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:43.621398 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:43.621451 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:43.621601 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:43.621602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:43.621762 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:43.621810 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:43.621811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.622491 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.622492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:43.623085 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:43.623087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.623543 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.623545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:43.623597 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:43.624144 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:43.624155 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:43.626427 (+ 2272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":109,"threads_started":1}
W20260620 18:39:43.626709 20386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:43.621322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:60753 (local address 127.10.250.126:38519)
0620 18:39:43.621466 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:43.621467 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:43.621477 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:39:43.621504 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:43.621504 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:43.621515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:43.621586 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:43.621894 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.622416 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.623169 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.623456 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.623548 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:43.626630 (+ 3082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:60753: BlockingRecv error: recv got EOF from 127.10.250.65:60753 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
Jun 20 18:39:43 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:43 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:43.766923 20387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:43.762023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:56267)
0620 18:39:43.762158 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:43.762168 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:43.762233 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:43.762353 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:43.762354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:43.762564 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:43.762633 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:43.762635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.763361 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.763362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:43.763891 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:43.763893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.764291 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.764291 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:43.764330 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:43.764795 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:43.764800 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:43.766821 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
W20260620 18:39:43.767032 20386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:43.762113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:56267 (local address 127.10.250.126:38519)
0620 18:39:43.762164 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:43.762166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:43.762221 (+ 55us) server_negotiation.cc:407] Connection header received
0620 18:39:43.762280 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:43.762280 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:43.762291 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:43.762340 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:43.762736 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.763241 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.763988 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.764183 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.764281 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:43.766947 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:56267: BlockingRecv error: recv got EOF from 127.10.250.66:56267 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 20 18:39:43 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:43 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:43.899986 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:43.894906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:46697)
0620 18:39:43.895032 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:43.895063 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:43.895129 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:43.895266 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:43.895267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:43.895455 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:43.895500 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:43.895502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.896418 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.896420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:43.896944 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:43.896946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.897371 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.897372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:43.897430 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:43.897869 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:43.897873 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:43.899926 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
W20260620 18:39:43.900120 20386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:43.895027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:46697 (local address 127.10.250.126:38519)
0620 18:39:43.895096 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:43.895098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:43.895112 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:39:43.895199 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:43.895200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:43.895212 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:43.895263 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:43.895592 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.896320 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.897062 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.897298 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.897368 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:43.900038 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:46697: BlockingRecv error: recv got EOF from 127.10.250.67:46697 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 18:39:44 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:44 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:44.635917 20389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:44.627160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:43803)
0620 18:39:44.627299 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:44.627311 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:44.627374 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:44.627627 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:44.627629 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:44.627803 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:44.627854 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:44.627856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:44.628579 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:44.628580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:44.629120 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:44.629121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:44.629588 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:44.629590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:44.629664 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:44.633599 (+ 3935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:44.633616 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:44.635853 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
W20260620 18:39:44.636094 20390 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:44.627240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:43803 (local address 127.10.250.126:38519)
0620 18:39:44.627394 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:44.627396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:44.627407 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:39:44.627441 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:44.627442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:44.627453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:44.627521 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:44.627951 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:44.628484 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:44.629208 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:44.629495 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:44.629586 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:44.635990 (+ 6404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:43803: BlockingRecv error: recv got EOF from 127.10.250.65:43803 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
Jun 20 18:39:44 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:44 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:44.775508 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:44.769973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:54707)
0620 18:39:44.770127 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:44.770138 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:44.770226 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:44.770473 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:44.770475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:44.770699 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:44.770748 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:44.770750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:44.771500 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:44.771501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:44.772048 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:44.772049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:44.772512 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:44.772513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:44.772569 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:44.773275 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:44.773281 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:44.775444 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
W20260620 18:39:44.775686 20390 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:44.770080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:54707 (local address 127.10.250.126:38519)
0620 18:39:44.770142 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:44.770143 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:44.770273 (+ 130us) server_negotiation.cc:407] Connection header received
0620 18:39:44.770307 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:44.770309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:44.770346 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:44.770420 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:44.770855 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:44.771413 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:44.772122 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:44.772422 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:44.772487 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:44.775613 (+ 3126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:54707: BlockingRecv error: recv got EOF from 127.10.250.66:54707 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 18:39:44 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:44 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:44.905148 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:44.900460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:34425)
0620 18:39:44.900610 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:44.900620 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:44.900677 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:44.900846 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:44.900848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:44.901034 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:44.901093 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:44.901094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:44.901808 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:44.901809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:44.902349 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:44.902351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:44.902720 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:44.902721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:44.902778 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:44.903215 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:44.903219 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:44.905085 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
W20260620 18:39:44.905272 20390 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:44.900536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:34425 (local address 127.10.250.126:38519)
0620 18:39:44.900589 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:44.900591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:44.900666 (+ 75us) server_negotiation.cc:407] Connection header received
0620 18:39:44.900736 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:44.900737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:44.900748 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:44.900793 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:44.901176 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:44.901707 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:44.902422 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:44.902653 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:44.902703 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:44.905174 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:34425: BlockingRecv error: recv got EOF from 127.10.250.67:34425 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 18:39:45 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:45 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:45.641587 20393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:45.636641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:38033)
0620 18:39:45.636842 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:45.636856 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:45.636919 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:45.637118 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:45.637119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:45.637330 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:45.637397 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:45.637400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:45.638142 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:45.638143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:45.638769 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:45.638771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:45.639204 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:45.639205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:45.639248 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:45.639677 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:45.639682 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:45.641526 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
W20260620 18:39:45.641721 20394 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:45.636708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:38033 (local address 127.10.250.126:38519)
0620 18:39:45.636874 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:45.636876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:45.636903 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:39:45.636978 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:45.636979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:45.636993 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:45.637101 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:45.637473 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:45.638049 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:45.638847 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:45.639109 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:45.639198 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:45.641630 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:38033: BlockingRecv error: recv got EOF from 127.10.250.65:38033 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
Jun 20 18:39:45 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:45 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:45.780867 20395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:45.776011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:40467)
0620 18:39:45.776185 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:45.776199 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:45.776328 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:45.776481 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:45.776482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:45.776664 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:45.776711 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:45.776712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:45.777392 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:45.777394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:45.777932 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:45.777933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:45.778330 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:45.778331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:45.778392 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:45.778976 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:45.778982 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:45.780802 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":108,"threads_started":1}
W20260620 18:39:45.781019 20394 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:45.776131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:40467 (local address 127.10.250.126:38519)
0620 18:39:45.776206 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:45.776208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:45.776305 (+ 97us) server_negotiation.cc:407] Connection header received
0620 18:39:45.776400 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:45.776401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:45.776413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:45.776473 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:45.776792 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:45.777302 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:45.778023 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:45.778238 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:45.778314 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:45.780926 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:40467: BlockingRecv error: recv got EOF from 127.10.250.66:40467 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 18:39:45 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:45 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:45.910557 20396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:45.905744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:44575)
0620 18:39:45.905922 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:45.905937 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:45.906034 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:45.906272 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:45.906273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:45.906454 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:45.906501 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:45.906503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:45.907207 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:45.907209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:45.907747 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:45.907749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:45.908118 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:45.908118 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:45.908181 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:45.908669 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:45.908673 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:45.910432 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":77,"threads_started":1}
W20260620 18:39:45.910616 20394 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:45.905851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:44575 (local address 127.10.250.126:38519)
0620 18:39:45.905941 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:45.905943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:45.906016 (+ 73us) server_negotiation.cc:407] Connection header received
0620 18:39:45.906109 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:45.906111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:45.906127 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:45.906259 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:45.906615 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:45.907116 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:45.907835 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:45.908034 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:45.908117 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:45.910552 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:44575: BlockingRecv error: recv got EOF from 127.10.250.67:44575 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 18:39:46 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:46 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:46.647249 20397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:46.642137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:45427)
0620 18:39:46.642280 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:46.642316 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:46.642376 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:46.642558 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:46.642559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:46.642728 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:46.642777 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:46.642778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:46.643446 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:46.643446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:46.643957 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:46.643958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:46.644328 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:46.644329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:46.644371 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:46.644910 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:46.644915 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:46.647165 (+ 2250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
W20260620 18:39:46.647377 20398 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:46.642188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:45427 (local address 127.10.250.126:38519)
0620 18:39:46.642327 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:46.642329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:46.642372 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:39:46.642422 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:46.642422 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:46.642433 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:46.642499 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:46.642849 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:46.643380 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:46.644036 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:46.644262 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:46.644318 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:46.647293 (+ 2975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:45427: BlockingRecv error: recv got EOF from 127.10.250.65:45427 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
Jun 20 18:39:46 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:46 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:46.786383 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:46.781391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:45759)
0620 18:39:46.781574 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:46.781586 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:46.781658 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:46.781798 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:46.781798 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:46.781987 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:46.782033 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:46.782035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:46.782761 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:46.782763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:46.783294 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:46.783297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:46.783719 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:46.783720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:46.783780 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:46.784295 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:46.784306 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:46.786268 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
W20260620 18:39:46.786437 20398 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:46.781504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:45759 (local address 127.10.250.126:38519)
0620 18:39:46.781571 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:46.781574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:46.781638 (+ 64us) server_negotiation.cc:407] Connection header received
0620 18:39:46.781718 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:46.781719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:46.781731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:46.781790 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:46.782114 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:46.782681 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:46.783425 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:46.783633 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:46.783718 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:46.786372 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:45759: BlockingRecv error: recv got EOF from 127.10.250.66:45759 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 18:39:46 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:46 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:46.916299 20400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:46.911130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:40703)
0620 18:39:46.911329 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:46.911342 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:46.911444 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:46.911637 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:46.911638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:46.911886 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:46.911948 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:46.911951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:46.912680 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:46.912681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:46.913252 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:46.913254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:46.913619 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:46.913620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:46.913663 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:46.914171 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:46.914175 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:46.916201 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":138,"threads_started":1}
W20260620 18:39:46.916356 20398 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:46.911296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:40703 (local address 127.10.250.126:38519)
0620 18:39:46.911355 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:46.911358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:46.911460 (+ 102us) server_negotiation.cc:407] Connection header received
0620 18:39:46.911529 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:46.911530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:46.911548 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:46.911609 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:46.912035 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:46.912582 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:46.913359 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:46.913547 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:46.913602 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:46.916291 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:40703: BlockingRecv error: recv got EOF from 127.10.250.67:40703 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 18:39:47 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:47 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:47.652376 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:47.647707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:60831)
0620 18:39:47.647890 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:47.647901 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:47.647953 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:47.648116 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:47.648117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:47.648311 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:47.648361 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:47.648363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:47.649007 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:47.649008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:47.649574 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:47.649575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:47.649913 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:47.649914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:47.649955 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:47.650408 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:47.650413 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:47.652285 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
W20260620 18:39:47.652463 20402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:47.647815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:60831 (local address 127.10.250.126:38519)
0620 18:39:47.647935 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:47.647937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:47.647946 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:47.647996 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:47.647997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:47.648007 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:47.648063 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:47.648431 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:47.648937 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:47.649647 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:47.649847 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:47.649899 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:47.652390 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:60831: BlockingRecv error: recv got EOF from 127.10.250.65:60831 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
Jun 20 18:39:47 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:47 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:47.792068 20403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:47.786925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:60439)
0620 18:39:47.787063 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:47.787075 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:47.787147 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:47.787399 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:47.787400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:47.787664 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:47.787718 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:47.787721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:47.788489 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:47.788490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:47.789043 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:47.789045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:47.789444 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:47.789445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:47.789504 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:47.789976 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:47.789981 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:47.791974 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
W20260620 18:39:47.792129 20402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:47.787006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:60439 (local address 127.10.250.126:38519)
0620 18:39:47.787101 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:47.787103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:47.787130 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:39:47.787264 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:47.787265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:47.787280 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:47.787386 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:47.787810 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:47.788357 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:47.789146 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:47.789351 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:47.789444 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:47.792069 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:60439: BlockingRecv error: recv got EOF from 127.10.250.66:60439 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 18:39:47 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:47 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:47.922092 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:47.916933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:47153)
0620 18:39:47.917177 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:47.917191 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:47.917284 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:47.917459 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:47.917460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:47.917641 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:47.917695 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:47.917697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:47.918403 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:47.918404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:47.919127 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:47.919129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:47.919516 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:47.919517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:47.919575 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:47.920038 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:47.920043 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:47.921981 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":141,"threads_started":1}
W20260620 18:39:47.922202 20402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:47.917092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:47153 (local address 127.10.250.126:38519)
0620 18:39:47.917230 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:47.917233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:47.917263 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:39:47.917375 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:47.917376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:47.917393 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:47.917443 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:47.917783 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:47.918307 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:47.919219 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:47.919426 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:47.919514 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:47.922132 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:47153: BlockingRecv error: recv got EOF from 127.10.250.67:47153 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jun 20 18:39:48 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:48 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:48.657816 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:48.652878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:40795)
0620 18:39:48.653054 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:48.653068 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:48.653131 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:48.653267 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:48.653268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:48.653424 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:48.653472 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:48.653473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:48.654271 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:48.654272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:48.654871 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:48.654873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:48.655246 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:48.655247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:48.655288 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:48.655813 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:48.655818 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:48.657744 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":85,"threads_started":1}
W20260620 18:39:48.657889 20406 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:48.653001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:40795 (local address 127.10.250.126:38519)
0620 18:39:48.653115 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:48.653116 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:48.653124 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:39:48.653178 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:48.653178 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:48.653188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:48.653247 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:48.653596 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:48.654195 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:48.654952 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:48.655181 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:48.655238 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:48.657836 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:40795: BlockingRecv error: recv got EOF from 127.10.250.65:40795 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
Jun 20 18:39:48 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:48 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:48.798281 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:48.792784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:60577)
0620 18:39:48.792998 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:48.793013 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:48.793112 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:48.793347 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:48.793348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:48.793547 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:48.793594 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:48.793596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:48.794746 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:48.794747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:48.795311 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:48.795313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:48.795752 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:48.795753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:48.795815 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:48.796304 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:48.796308 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:48.798182 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":113,"threads_started":1}
W20260620 18:39:48.798331 20406 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:48.792914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:60577 (local address 127.10.250.126:38519)
0620 18:39:48.793002 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:48.793005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:48.793091 (+ 86us) server_negotiation.cc:407] Connection header received
0620 18:39:48.793241 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:48.793242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:48.793256 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:48.793327 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:48.793698 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:48.794668 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:48.795399 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:48.795643 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:48.795712 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:48.798266 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:60577: BlockingRecv error: recv got EOF from 127.10.250.66:60577 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 18:39:48 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:48 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:48.927696 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:48.922717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:45949)
0620 18:39:48.923040 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:48.923052 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:48.923131 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:48.923341 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:48.923341 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:48.923529 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:48.923578 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:48.923580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:48.924278 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:48.924279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:48.924811 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:48.924812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:48.925174 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:48.925175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:48.925233 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:48.925732 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:48.925737 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:48.927586 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":113,"threads_started":1}
W20260620 18:39:48.927758 20406 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:48.922833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:45949 (local address 127.10.250.126:38519)
0620 18:39:48.923015 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:48.923017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:48.923128 (+ 111us) server_negotiation.cc:407] Connection header received
0620 18:39:48.923236 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:48.923237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:48.923259 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:48.923329 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:48.923706 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:48.924190 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:48.924895 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:48.925082 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:48.925167 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:48.927696 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:45949: BlockingRecv error: recv got EOF from 127.10.250.67:45949 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
Jun 20 18:39:49 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:49 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:49.667305 20409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:49.658376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:48465)
0620 18:39:49.658631 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:49.658643 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:49.658712 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:49.658850 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:49.658850 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:49.659021 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:49.659070 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:49.659071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:49.659731 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:49.659732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:49.660280 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:49.660281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:49.660644 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:49.660645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:49.660684 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:49.665351 (+ 4667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:49.665356 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:49.667221 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":123,"threads_started":1}
W20260620 18:39:49.667514 20410 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:49.658499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:48465 (local address 127.10.250.126:38519)
0620 18:39:49.658658 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:49.658659 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:49.658694 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:39:49.658758 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:49.658759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:49.658769 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:49.658829 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:49.659136 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:49.659658 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:49.660367 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:49.660570 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:49.660638 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:49.667406 (+ 6768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:48465: BlockingRecv error: recv got EOF from 127.10.250.65:48465 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
Jun 20 18:39:49 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:49 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:49.804107 20411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:49.798825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:58175)
0620 18:39:49.799005 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:49.799018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:49.799107 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:49.799312 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:49.799313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:49.799504 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:49.799547 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:49.799549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:49.800239 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:49.800240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:49.800772 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:49.800773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:49.801148 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:49.801149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:49.801209 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:49.801718 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:49.801722 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:49.804030 (+ 2308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
W20260620 18:39:49.804270 20410 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:49.798916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:58175 (local address 127.10.250.126:38519)
0620 18:39:49.798992 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:49.798995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:49.799086 (+ 91us) server_negotiation.cc:407] Connection header received
0620 18:39:49.799230 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:49.799231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:49.799245 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:49.799304 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:49.799655 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:49.800154 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:49.800857 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:49.801071 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:49.801149 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:49.804171 (+ 3022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:58175: BlockingRecv error: recv got EOF from 127.10.250.66:58175 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 18:39:49 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:49 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:49.932981 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:49.928294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:49299)
0620 18:39:49.928448 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:49.928459 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:49.928514 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:49.928634 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:49.928635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:49.928809 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:49.928860 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:49.928861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:49.929566 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:49.929567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:49.930125 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:49.930126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:49.930583 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:49.930584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:49.930634 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:49.931114 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:49.931119 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:49.932919 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
W20260620 18:39:49.933171 20410 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:49.928401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:49299 (local address 127.10.250.126:38519)
0620 18:39:49.928456 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:49.928458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:49.928501 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:39:49.928560 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:49.928560 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:49.928572 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:49.928621 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:49.928956 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:49.929475 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:49.930232 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:49.930444 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:49.930534 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:49.933090 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:49299: BlockingRecv error: recv got EOF from 127.10.250.67:49299 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 18:39:50 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:50 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:50.673171 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:50.667851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:48237)
0620 18:39:50.668052 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:50.668064 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:50.668127 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:50.668285 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:50.668286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:50.668450 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:50.668499 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:50.668501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:50.669308 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:50.669310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:50.670007 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:50.670010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:50.670469 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:50.670470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:50.670572 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:50.671132 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:50.671137 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:50.673059 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":107,"threads_started":1}
W20260620 18:39:50.673250 20414 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:50.667976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:48237 (local address 127.10.250.126:38519)
0620 18:39:50.668118 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:50.668120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:50.668130 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:39:50.668193 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:50.668194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:50.668207 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:50.668258 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:50.668619 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:50.669192 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:50.670093 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:50.670387 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:50.670443 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:50.673172 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:48237: BlockingRecv error: recv got EOF from 127.10.250.65:48237 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
Jun 20 18:39:50 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:50 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:50.809521 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:50.804621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:59049)
0620 18:39:50.804792 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:50.804804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:50.804899 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:50.805054 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:50.805054 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:50.805241 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:50.805286 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:50.805288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:50.805966 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:50.805967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:50.806546 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:50.806549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:50.806945 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:50.806946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:50.807013 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:50.807505 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:50.807510 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:50.809461 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":127,"threads_started":1}
W20260620 18:39:50.809716 20414 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:50.804709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:59049 (local address 127.10.250.126:38519)
0620 18:39:50.804790 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:50.804792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:50.804874 (+ 82us) server_negotiation.cc:407] Connection header received
0620 18:39:50.804988 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:50.804989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:50.805000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:50.805048 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:50.805367 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:50.805880 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:50.806643 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:50.806858 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:50.806945 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:50.809653 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:59049: BlockingRecv error: recv got EOF from 127.10.250.66:59049 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 18:39:50 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:50 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:50.938802 20416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:50.933639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:58185)
0620 18:39:50.933865 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:50.933879 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:50.933953 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:50.934111 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:50.934112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:50.934290 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:50.934338 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:50.934339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:50.935103 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:50.935104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:50.935615 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:50.935616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:50.936036 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:50.936037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:50.936100 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:50.936697 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:50.936703 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:50.938730 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":103,"threads_started":1}
W20260620 18:39:50.938992 20414 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:50.933852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:58185 (local address 127.10.250.126:38519)
0620 18:39:50.933928 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:50.933929 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:50.933940 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:39:50.934019 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:50.934020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:50.934034 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:50.934087 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:50.934421 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:50.935033 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:50.935720 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:50.935964 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:50.936042 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:50.938927 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:58185: BlockingRecv error: recv got EOF from 127.10.250.67:58185 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 18:39:51 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:51 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:51.678659 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:51.673787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:49111)
0620 18:39:51.673946 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:51.673959 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:51.674022 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:51.674201 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:51.674202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:51.674369 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:51.674418 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:51.674419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:51.675188 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:51.675189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:51.675734 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:51.675735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:51.676137 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:51.676138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:51.676188 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:51.676642 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:51.676648 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:51.678590 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
W20260620 18:39:51.678838 20417 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:51.673778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:49111 (local address 127.10.250.126:38519)
0620 18:39:51.673906 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:51.673908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:51.674018 (+ 110us) server_negotiation.cc:407] Connection header received
0620 18:39:51.674085 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:51.674086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:51.674097 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:51.674154 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:51.674530 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:51.675054 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:51.675815 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:51.676013 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:51.676080 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:51.678748 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:49111: BlockingRecv error: recv got EOF from 127.10.250.65:49111 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
Jun 20 18:39:51 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:51 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:51.815208 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:51.810070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:51861)
0620 18:39:51.810240 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:51.810254 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:51.810326 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:51.810466 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:51.810467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:51.810682 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:51.810738 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:51.810740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:51.811532 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:51.811534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:51.812097 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:51.812098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:51.812523 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:51.812524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:51.812588 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:51.813155 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:51.813160 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:51.815131 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
W20260620 18:39:51.815397 20417 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:51.810203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:51861 (local address 127.10.250.126:38519)
0620 18:39:51.810284 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:51.810286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:51.810308 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:39:51.810390 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:51.810391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:51.810402 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:51.810443 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:51.810845 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:51.811421 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:51.812211 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:51.812415 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:51.812474 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:51.815333 (+ 2859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:51861: BlockingRecv error: recv got EOF from 127.10.250.66:51861 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 18:39:51 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:51 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:51.946590 20420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:51.939317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:60697)
0620 18:39:51.939517 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:51.939529 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:51.939592 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:51.940337 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:51.940339 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:51.940583 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:51.940649 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:51.940651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:51.941568 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:51.941570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:51.942478 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:51.942480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:51.942954 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:51.942955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:51.943032 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:51.943662 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:51.943667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:51.946441 (+ 2774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":112,"threads_started":1}
W20260620 18:39:51.946797 20417 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:51.939352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:60697 (local address 127.10.250.126:38519)
0620 18:39:51.939401 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:51.939403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:51.940196 (+ 793us) server_negotiation.cc:407] Connection header received
0620 18:39:51.940230 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:51.940231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:51.940246 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:51.940307 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:51.940752 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:51.941470 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:51.942627 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:51.942857 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:51.942952 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:51.946589 (+ 3637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:60697: BlockingRecv error: recv got EOF from 127.10.250.67:60697 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 18:39:52 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:52 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:52.684294 20421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:52.679315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:53713)
0620 18:39:52.679468 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:52.679486 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:52.679539 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:52.679738 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:52.679740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:52.679910 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:52.679957 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:52.679958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:52.680725 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:52.680726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:52.681258 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:52.681259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:52.681685 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:52.681686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:52.681744 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:52.682285 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:52.682290 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:52.684233 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
W20260620 18:39:52.684432 20422 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:52.679399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:53713 (local address 127.10.250.126:38519)
0620 18:39:52.679579 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:52.679581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:52.679590 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:52.679612 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:52.679613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:52.679623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:52.679691 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:52.680100 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:52.680617 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:52.681365 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:52.681587 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:52.681655 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:52.684340 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:53713: BlockingRecv error: recv got EOF from 127.10.250.65:53713 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
Jun 20 18:39:52 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:52 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:52.820793 20423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:52.815897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:34509)
0620 18:39:52.816080 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:52.816090 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:52.816152 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:52.816326 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:52.816326 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:52.816510 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:52.816563 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:52.816565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:52.817277 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:52.817279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:52.817835 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:52.817837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:52.818227 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:52.818229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:52.818279 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:52.818866 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:52.818870 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:52.820698 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
W20260620 18:39:52.820868 20422 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:52.816013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:34509 (local address 127.10.250.126:38519)
0620 18:39:52.816121 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:52.816123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:52.816200 (+ 77us) server_negotiation.cc:407] Connection header received
0620 18:39:52.816228 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:52.816229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:52.816241 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:52.816295 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:52.816660 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:52.817179 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:52.817926 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:52.818113 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:52.818197 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:52.820803 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:34509: BlockingRecv error: recv got EOF from 127.10.250.66:34509 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 18:39:52 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:52 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:52.951869 20424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:52.947157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:59975)
0620 18:39:52.947308 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:52.947319 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:52.947372 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:52.947570 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:52.947571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:52.947751 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:52.947800 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:52.947802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:52.948520 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:52.948521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:52.949086 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:52.949088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:52.949437 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:52.949438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:52.949479 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:52.950023 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:52.950028 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:52.951799 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
W20260620 18:39:52.952000 20422 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:52.947270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:59975 (local address 127.10.250.126:38519)
0620 18:39:52.947324 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:52.947326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:52.947361 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:39:52.947426 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:52.947427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:52.947439 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:52.947497 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:52.947904 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:52.948434 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:52.949179 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:52.949368 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:52.949428 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:52.951899 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:59975: BlockingRecv error: recv got EOF from 127.10.250.67:59975 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 18:39:53 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:53 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:53.689792 20425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:53.684817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:57547)
0620 18:39:53.684984 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:53.684999 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:53.685060 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:53.685221 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:53.685222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:53.685403 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:53.685451 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:53.685452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:53.686131 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:53.686161 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:53.686772 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:53.686774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:53.687214 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:53.687215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:53.687271 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:53.687847 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:53.687852 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:53.689707 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
W20260620 18:39:53.689895 20426 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:53.684865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:57547 (local address 127.10.250.126:38519)
0620 18:39:53.685004 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:53.685006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:53.685046 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:39:53.685120 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:53.685121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:53.685131 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:53.685210 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:53.685548 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:53.686048 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:53.686882 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:53.687088 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:53.687163 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:53.689819 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:57547: BlockingRecv error: recv got EOF from 127.10.250.65:57547 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
Jun 20 18:39:53 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:53 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:53.826139 20427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:53.821387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:60933)
0620 18:39:53.821534 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:53.821546 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:53.821606 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:53.821756 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:53.821757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:53.821922 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:53.821972 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:53.821974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:53.822696 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:53.822697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:53.823283 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:53.823286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:53.823658 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:53.823658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:53.823708 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:53.824205 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:53.824214 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:53.826077 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
W20260620 18:39:53.826223 20426 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:53.821524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:60933 (local address 127.10.250.126:38519)
0620 18:39:53.821625 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:53.821627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:53.821636 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:53.821662 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:53.821663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:53.821673 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:53.821734 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:53.822071 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:53.822616 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:53.823379 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:53.823569 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:53.823638 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:53.826176 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:60933: BlockingRecv error: recv got EOF from 127.10.250.66:60933 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 18:39:53 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:53 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:53.957036 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:53.952374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:47899)
0620 18:39:53.952551 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:53.952562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:53.952628 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:53.952754 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:53.952754 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:53.952918 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:53.952965 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:53.952967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:53.953576 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:53.953577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:53.954205 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:53.954207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:53.954620 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:53.954621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:53.954705 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:53.955206 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:53.955219 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:53.956977 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
W20260620 18:39:53.957192 20426 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:53.952454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:47899 (local address 127.10.250.126:38519)
0620 18:39:53.952555 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:53.952557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:53.952612 (+ 55us) server_negotiation.cc:407] Connection header received
0620 18:39:53.952671 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:53.952672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:53.952684 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:53.952736 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:53.953026 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:53.953506 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:53.954278 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:53.954491 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:53.954601 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:53.957107 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:47899: BlockingRecv error: recv got EOF from 127.10.250.67:47899 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 18:39:54 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:54 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:54.698868 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:54.690398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:33755)
0620 18:39:54.690587 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:54.690606 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:54.690656 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:54.690817 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:54.690817 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:54.690990 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:54.691044 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:54.691046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:54.691802 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:54.691803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:54.692392 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:54.692395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:54.692817 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:54.692818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:54.692859 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:54.696724 (+ 3865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:54.696729 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:54.698806 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
W20260620 18:39:54.699019 20430 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:54.690453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:33755 (local address 127.10.250.126:38519)
0620 18:39:54.690625 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:54.690627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:54.690639 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:39:54.690721 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:54.690721 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:54.690733 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:54.690809 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:54.691120 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:54.691704 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:54.692508 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:54.692731 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:54.692812 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:54.698932 (+ 6120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:33755: BlockingRecv error: recv got EOF from 127.10.250.65:33755 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
Jun 20 18:39:54 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:54 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:54.831931 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:54.826736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:33787)
0620 18:39:54.826911 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:54.826922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:54.826987 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:54.827118 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:54.827119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:54.827281 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:54.827326 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:54.827328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:54.827979 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:54.827980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:54.828560 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:54.828562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:54.829074 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:54.829075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:54.829125 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:54.829712 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:54.829717 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:54.831862 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
W20260620 18:39:54.832082 20430 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:54.826862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:33787 (local address 127.10.250.126:38519)
0620 18:39:54.826922 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:54.826924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:54.826995 (+ 71us) server_negotiation.cc:407] Connection header received
0620 18:39:54.827042 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:54.827043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:54.827053 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:54.827103 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:54.827397 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:54.827900 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:54.828672 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:54.828894 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:54.829048 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:54.832001 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:33787: BlockingRecv error: recv got EOF from 127.10.250.66:33787 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 18:39:54 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:54 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.67@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:54.962304 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:54.957512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.67:35755)
0620 18:39:54.957707 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:54.957718 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:54.957783 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:54.958026 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:54.958027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:54.958193 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:54.958239 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:54.958240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:54.958971 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:54.958972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:54.959532 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:54.959534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:54.959889 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:54.959889 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:54.959928 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:54.960395 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:54.960401 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:54.962233 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":95,"threads_started":1}
W20260620 18:39:54.962428 20430 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:54.957644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:35755 (local address 127.10.250.126:38519)
0620 18:39:54.957704 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:54.957706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:54.957788 (+ 82us) server_negotiation.cc:407] Connection header received
0620 18:39:54.957898 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:54.957899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:54.957916 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:54.958006 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:54.958320 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:54.958887 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:54.959639 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:54.959823 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:54.959877 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:54.962345 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.67:35755: BlockingRecv error: recv got EOF from 127.10.250.67:35755 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 18:39:55 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:55 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.65@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:55.704869 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:55.699484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.65:46689)
0620 18:39:55.699672 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:55.699684 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:55.699744 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:55.699915 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:55.699916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:55.700110 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:55.700161 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:55.700162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:55.700820 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:55.700821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:55.701546 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:55.701549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:55.701912 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:55.701912 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:55.701954 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:55.702396 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:55.702401 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:55.704758 (+ 2357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":102,"threads_started":1}
W20260620 18:39:55.704913 20434 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:55.699602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:46689 (local address 127.10.250.126:38519)
0620 18:39:55.699730 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:55.699732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:55.699741 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:55.699793 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:55.699793 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:55.699803 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:55.699862 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:55.700232 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:55.700732 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:55.701643 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:55.701843 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:55.701900 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:55.704839 (+ 2939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.65:46689: BlockingRecv error: recv got EOF from 127.10.250.65:46689 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
Jun 20 18:39:55 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:55 dist-test-slave-wp6g krb5kdc[19375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.250.66@KRBTEST.COM for oryx/127.10.250.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:55.837976 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:55.832668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38519 (local address 127.10.250.66:58633)
0620 18:39:55.832916 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:55.832929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:55.833049 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:55.833232 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:55.833233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:55.833415 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:55.833462 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:55.833464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:55.834189 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:55.834190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:55.834749 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:55.834751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:55.835130 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:55.835131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:55.835190 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:55.835802 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:55.835825 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:55.837911 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.250.126:38519: Server oryx/127.10.250.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":87,"threads_started":1}
W20260620 18:39:55.838167 20434 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:55.832797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:58633 (local address 127.10.250.126:38519)
0620 18:39:55.832899 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:55.832901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:55.833026 (+ 125us) server_negotiation.cc:407] Connection header received
0620 18:39:55.833128 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:55.833129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:55.833147 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:55.833210 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:55.833549 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:55.834084 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:55.834834 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:55.835036 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:55.835127 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:55.838084 (+ 2957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.250.66:58633: BlockingRecv error: recv got EOF from 127.10.250.66:58633 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:39:55.879580 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 19958
I20260620 18:39:55.884260 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 20110
I20260620 18:39:55.888586 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 20244
I20260620 18:39:55.892814 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 19888
2026-06-20T18:39:55Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17538 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T18:39:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:55Z Disabled control of system clock
I20260620 18:39:55.920853 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:36525
--webserver_interface=127.10.250.126
--webserver_port=0
--builtin_ntp_servers=127.10.250.84:41205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:36525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 18:39:56.003322 20439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:56.003564 20439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:56.003587 20439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:56.004879 20439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:56.004925 20439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:56.004938 20439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:56.004951 20439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:56.004963 20439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:56.006345 20439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:36525
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:36525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:56.006651 20439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:56.006920 20439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:39:56.012346 20439 server_base.cc:1034] running on GCE node
W20260620 18:39:56.012612 20445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:56.012665 20444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:56.012714 20447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:56.013334 20439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:56.013675 20439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:56.014926 20439 hybrid_clock.cc:648] HybridClock initialized: now 1781980796014899 us; error 49 us; skew 500 ppm
I20260620 18:39:56.016505 20439 webserver.cc:460] Webserver started at http://127.10.250.126:38177/ using document root <none> and password file <none>
I20260620 18:39:56.016793 20439 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:56.016875 20439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:56.017050 20439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:56.018278 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "525efcc533874043af3f2d4129a09739"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "d9547981b022186f67ccaf1513dae638"
server_key_iv: "3e868af865cdfc28c2a4714a5e5f9918"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.018718 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "525efcc533874043af3f2d4129a09739"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "d9547981b022186f67ccaf1513dae638"
server_key_iv: "3e868af865cdfc28c2a4714a5e5f9918"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.020260 20439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:56.021246 20453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.021499 20439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:56.021633 20439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "525efcc533874043af3f2d4129a09739"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "d9547981b022186f67ccaf1513dae638"
server_key_iv: "3e868af865cdfc28c2a4714a5e5f9918"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.021721 20439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:56.042294 20439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:56.042722 20439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:56.042912 20439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:56.047617 20439 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:36525
I20260620 18:39:56.047654 20505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:36525 every 8 connection(s)
I20260620 18:39:56.048028 20439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:56.049059 20506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:56.051497 20506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739: Bootstrap starting.
I20260620 18:39:56.052162 20506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:56.052482 20506 log.cc:824] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:56.053288 20506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739: No bootstrap required, opened a new log
I20260620 18:39:56.055032 20506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525efcc533874043af3f2d4129a09739" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36525 } }
I20260620 18:39:56.055186 20506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:56.055209 20506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525efcc533874043af3f2d4129a09739, State: Initialized, Role: FOLLOWER
I20260620 18:39:56.055363 20506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525efcc533874043af3f2d4129a09739" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36525 } }
I20260620 18:39:56.055449 20506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:56.055471 20506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:56.055498 20506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:56.055680 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 20439
I20260620 18:39:56.055770 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:56.055940 11241 external_mini_cluster.cc:1428] Setting key f37e53ab9a0832454de6853f39f0cc12
I20260620 18:39:56.056167 20506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525efcc533874043af3f2d4129a09739" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36525 } }
I20260620 18:39:56.056257 20506 leader_election.cc:302] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 525efcc533874043af3f2d4129a09739; no voters:
I20260620 18:39:56.056471 20506 leader_election.cc:288] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:56.056531 20509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:56.056715 20509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [term 1 LEADER]: Becoming Leader. State: Replica: 525efcc533874043af3f2d4129a09739, State: Running, Role: LEADER
I20260620 18:39:56.056797 20509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525efcc533874043af3f2d4129a09739" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36525 } }
I20260620 18:39:56.056834 20506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:56.057106 20509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "525efcc533874043af3f2d4129a09739" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525efcc533874043af3f2d4129a09739" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36525 } } }
I20260620 18:39:56.057175 20509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:56.057305 20509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 525efcc533874043af3f2d4129a09739. Latest consensus state: current_term: 1 leader_uuid: "525efcc533874043af3f2d4129a09739" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525efcc533874043af3f2d4129a09739" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36525 } } }
I20260620 18:39:56.057443 20509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:56.057540 20515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:56.058223 20515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:56.059909 20515 catalog_manager.cc:1347] Generated new cluster ID: 5198906d6f804d70990aa2295fe422d2
I20260620 18:39:56.059962 20515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:56.063970 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:56.056565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.10.250.126:36525)
0620 18:39:56.058303 (+ 1738us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:56.058312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:56.058350 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:39:56.058551 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:56.058559 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:56.058639 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:56.058799 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:56.059032 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:56.060288 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:56.061865 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:56.062369 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:56.062514 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:56.063306 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:56.063310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:56.063311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:56.063496 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:56.063538 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:56.063550 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:56.063629 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:56.063706 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 18:39:56.063787 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1649,"thread_start_us":110,"threads_started":1}
I20260620 18:39:56.074283 20515 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:56.074928 20515 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:56.081704 20515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 525efcc533874043af3f2d4129a09739: Generated new TSK 0
I20260620 18:39:56.081943 20515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:39:56.085114 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:36525
--builtin_ntp_servers=127.10.250.84:41205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 18:39:56.179198 20530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:56.179538 20530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:56.179587 20530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:56.181358 20530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:56.181416 20530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:56.181447 20530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:56.183187 20530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:36525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:56.183472 20530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:56.183816 20530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:56.189203 20536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:56.189253 20538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:56.189306 20535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:56.189496 20530 server_base.cc:1034] running on GCE node
I20260620 18:39:56.189721 20530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:56.189998 20530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:56.191217 20530 hybrid_clock.cc:648] HybridClock initialized: now 1781980796191199 us; error 30 us; skew 500 ppm
I20260620 18:39:56.192723 20530 webserver.cc:460] Webserver started at http://127.10.250.65:42377/ using document root <none> and password file <none>
I20260620 18:39:56.192987 20530 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:56.193035 20530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:56.193195 20530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:56.194345 20530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "fc00636c7c004d6380c1c72a581458f9"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "3d026e39da089c3c0e7e87a61bd9edb7"
server_key_iv: "db8aea151d2d703cec5ae048421dee22"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.194753 20530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "fc00636c7c004d6380c1c72a581458f9"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "3d026e39da089c3c0e7e87a61bd9edb7"
server_key_iv: "db8aea151d2d703cec5ae048421dee22"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.196208 20530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:56.197021 20544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.197187 20530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:56.197309 20530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "fc00636c7c004d6380c1c72a581458f9"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "3d026e39da089c3c0e7e87a61bd9edb7"
server_key_iv: "db8aea151d2d703cec5ae048421dee22"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.197394 20530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:56.225672 20530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:56.226046 20530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:56.226202 20530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:56.226475 20530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:56.226869 20530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:56.226904 20530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.226926 20530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:56.226939 20530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.232913 20530 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:35655
I20260620 18:39:56.232964 20657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:35655 every 8 connection(s)
I20260620 18:39:56.233357 20530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:56.234464 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 20530
I20260620 18:39:56.234602 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:56.234803 11241 external_mini_cluster.cc:1428] Setting key 17284413f022b6162454ad8c31f3c79d
I20260620 18:39:56.235790 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:36525
--builtin_ntp_servers=127.10.250.84:41205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 18:39:56.239470 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:56.234234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:37761 (local address 127.10.250.126:36525)
0620 18:39:56.234303 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:56.234305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:56.234616 (+ 311us) server_negotiation.cc:407] Connection header received
0620 18:39:56.235324 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:56.235325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:56.235336 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:56.235424 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:56.236242 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:56.236778 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:56.237930 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:56.238128 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:56.238186 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:56.239181 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:56.239182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:56.239183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:56.239188 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:56.239205 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:56.239209 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:56.239264 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:56.239375 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 18:39:56.239408 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:39:56.239804 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:56.234227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:36525 (local address 127.10.250.65:37761)
0620 18:39:56.234434 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:56.234470 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:56.235225 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:56.235511 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:56.235516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:56.235527 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:56.236142 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:56.236149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:56.236898 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:56.236900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:56.237837 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:56.237842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:56.238220 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:56.238221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:56.238389 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:56.238989 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:56.238990 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:56.239044 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:56.239079 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:56.239284 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:56.239291 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:56.239295 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:39:56.239376 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 18:39:56.239507 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
I20260620 18:39:56.240372 20658 heartbeater.cc:347] Connected to a master server at 127.10.250.126:36525
I20260620 18:39:56.240470 20658 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:56.240675 20658 heartbeater.cc:511] Master 127.10.250.126:36525 requested a full tablet report, sending...
I20260620 18:39:56.241237 20470 ts_manager.cc:194] Registered new tserver with Master: fc00636c7c004d6380c1c72a581458f9 (127.10.250.65:35655)
I20260620 18:39:56.241858 20470 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.250.65:37761
W20260620 18:39:56.318863 20661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:56.319109 20661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:56.319149 20661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:56.320317 20661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:56.320379 20661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:56.320421 20661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:56.321750 20661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:36525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:56.322055 20661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:56.322367 20661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:56.327582 20667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:56.327702 20669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:56.327582 20666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:56.327821 20661 server_base.cc:1034] running on GCE node
I20260620 18:39:56.328068 20661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:56.328400 20661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:56.329579 20661 hybrid_clock.cc:648] HybridClock initialized: now 1781980796329574 us; error 46 us; skew 500 ppm
I20260620 18:39:56.331044 20661 webserver.cc:460] Webserver started at http://127.10.250.66:42027/ using document root <none> and password file <none>
I20260620 18:39:56.331336 20661 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:56.331385 20661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:56.331549 20661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:56.332629 20661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "9ce2364b35a14b3c87bbf8a52cfdb547"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "90b1f567665ebfe4bfab6ef4eef06df5"
server_key_iv: "b2a8d8a7d562b5539648f31ffbb105b8"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.332988 20661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "9ce2364b35a14b3c87bbf8a52cfdb547"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "90b1f567665ebfe4bfab6ef4eef06df5"
server_key_iv: "b2a8d8a7d562b5539648f31ffbb105b8"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.334692 20661 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 18:39:56.335523 20675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.335695 20661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:56.335793 20661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "9ce2364b35a14b3c87bbf8a52cfdb547"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "90b1f567665ebfe4bfab6ef4eef06df5"
server_key_iv: "b2a8d8a7d562b5539648f31ffbb105b8"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.335881 20661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:56.347519 20661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:56.347857 20661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:56.348009 20661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:56.348250 20661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:56.348583 20661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:56.348639 20661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.348683 20661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:56.348706 20661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.354429 20661 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:38015
I20260620 18:39:56.354559 20788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:38015 every 8 connection(s)
I20260620 18:39:56.354876 20661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:56.360348 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:56.355662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:32831 (local address 127.10.250.126:36525)
0620 18:39:56.355730 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:56.355732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:56.355982 (+ 250us) server_negotiation.cc:407] Connection header received
0620 18:39:56.356496 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:56.356498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:56.356508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:56.356573 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:56.357361 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:56.357914 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:56.359025 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:56.359220 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:56.359277 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:56.360117 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:56.360118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:56.360119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:56.360124 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:56.360138 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:56.360142 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:56.360185 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:56.360264 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 18:39:56.360287 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:39:56.360555 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:56.355649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:36525 (local address 127.10.250.66:32831)
0620 18:39:56.355855 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:56.355896 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:56.356382 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:56.356627 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:56.356631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:56.356639 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:56.357276 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:56.357282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:56.358015 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:56.358016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:56.358954 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:56.358956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:56.359297 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:56.359298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:56.359412 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:56.359924 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:56.359924 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:56.359954 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:56.359976 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:56.360198 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:56.360203 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:56.360205 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:39:56.360257 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 18:39:56.360337 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
I20260620 18:39:56.361121 20789 heartbeater.cc:347] Connected to a master server at 127.10.250.126:36525
I20260620 18:39:56.361239 20789 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:56.361205 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 20661
I20260620 18:39:56.361305 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:56.361433 20789 heartbeater.cc:511] Master 127.10.250.126:36525 requested a full tablet report, sending...
I20260620 18:39:56.361490 11241 external_mini_cluster.cc:1428] Setting key ba9bdf4d4c7495ce958144dec4da47df
I20260620 18:39:56.361850 20470 ts_manager.cc:194] Registered new tserver with Master: 9ce2364b35a14b3c87bbf8a52cfdb547 (127.10.250.66:38015)
I20260620 18:39:56.362313 20470 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.250.66:32831
I20260620 18:39:56.362695 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:36525
--builtin_ntp_servers=127.10.250.84:41205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 18:39:56.444952 20792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:56.445207 20792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:56.445245 20792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:56.446434 20792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:56.446498 20792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:56.446565 20792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:56.447916 20792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:36525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:56.448180 20792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:56.448478 20792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:39:56.453809 20792 server_base.cc:1034] running on GCE node
W20260620 18:39:56.453779 20797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:56.453779 20800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:56.453801 20798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:56.454244 20792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:56.454509 20792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:56.455721 20792 hybrid_clock.cc:648] HybridClock initialized: now 1781980796455710 us; error 27 us; skew 500 ppm
I20260620 18:39:56.457302 20792 webserver.cc:460] Webserver started at http://127.10.250.67:35403/ using document root <none> and password file <none>
I20260620 18:39:56.457576 20792 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:56.457623 20792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:56.457775 20792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:56.458976 20792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "3e034b3a5c394050882b8e171619f42c"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "0a0822abe64f805c723e17b18d442e2b"
server_key_iv: "65aab3f1a5cbc44a45d203eddf09bcce"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.459345 20792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "3e034b3a5c394050882b8e171619f42c"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "0a0822abe64f805c723e17b18d442e2b"
server_key_iv: "65aab3f1a5cbc44a45d203eddf09bcce"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.460656 20792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 18:39:56.461524 20806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.461737 20792 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:56.461850 20792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "3e034b3a5c394050882b8e171619f42c"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "0a0822abe64f805c723e17b18d442e2b"
server_key_iv: "65aab3f1a5cbc44a45d203eddf09bcce"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.461927 20792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:56.482743 20792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:56.483084 20792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:56.483246 20792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:56.483505 20792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:56.483951 20792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:56.484009 20792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.484052 20792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:56.484081 20792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.489732 20792 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:46543
I20260620 18:39:56.489813 20919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:46543 every 8 connection(s)
I20260620 18:39:56.490157 20792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:56.495762 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:56.491066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:41599 (local address 127.10.250.126:36525)
0620 18:39:56.491145 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:56.491148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:56.491440 (+ 292us) server_negotiation.cc:407] Connection header received
0620 18:39:56.491948 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:56.491950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:56.491962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:56.492022 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:56.492758 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:56.493267 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:56.494357 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:56.494571 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:56.494629 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:56.495476 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:56.495477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:56.495477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:56.495482 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:56.495495 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:56.495499 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:56.495544 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:56.495672 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 18:39:56.495699 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:39:56.496028 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:56.491074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:36525 (local address 127.10.250.67:41599)
0620 18:39:56.491296 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:56.491332 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:56.491851 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:56.492135 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:56.492139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:56.492148 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:56.492663 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:56.492667 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:56.493373 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:56.493374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:56.494260 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:56.494263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:56.494667 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:56.494668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:56.494798 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:56.495339 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:56.495339 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:56.495370 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:56.495393 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:56.495582 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:56.495588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:56.495591 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:56.495665 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 18:39:56.495760 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
I20260620 18:39:56.496603 20920 heartbeater.cc:347] Connected to a master server at 127.10.250.126:36525
I20260620 18:39:56.496711 20920 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:56.496927 20920 heartbeater.cc:511] Master 127.10.250.126:36525 requested a full tablet report, sending...
I20260620 18:39:56.497362 20470 ts_manager.cc:194] Registered new tserver with Master: 3e034b3a5c394050882b8e171619f42c (127.10.250.67:46543)
I20260620 18:39:56.497778 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 20792
I20260620 18:39:56.497856 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:56.497901 20470 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.250.67:41599
I20260620 18:39:56.498049 11241 external_mini_cluster.cc:1428] Setting key 20220881cc65aa7658143d9ba76e0401
I20260620 18:39:56.498891 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:39:56.500844 20512 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:56.500303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.10.250.126:36525)
0620 18:39:56.500344 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:56.500345 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:56.500438 (+ 93us) server_negotiation.cc:407] Connection header received
0620 18:39:56.500474 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:56.500475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:56.500485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:56.500538 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:56.500778 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60560: BlockingRecv error: recv got EOF from 127.0.0.1:60560 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:39:56.500905 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 20530
I20260620 18:39:56.505707 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 20661
I20260620 18:39:56.510075 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 20792
I20260620 18:39:56.514384 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 20439
2026-06-20T18:39:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (620 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T18:39:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:56Z Disabled control of system clock
I20260620 18:39:56.541226 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:37395
--webserver_interface=127.10.250.126
--webserver_port=0
--builtin_ntp_servers=127.10.250.84:41017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:37395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:39:56.623224 20931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:56.623448 20931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:56.623471 20931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:56.624627 20931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:56.624666 20931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:56.624679 20931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:56.624692 20931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:56.624704 20931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:56.626111 20931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:37395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:37395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:56.626313 20931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:56.626652 20931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:56.631498 20939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:56.631548 20936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:56.631513 20937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:56.631757 20931 server_base.cc:1034] running on GCE node
I20260620 18:39:56.631982 20931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:56.632289 20931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:56.633440 20931 hybrid_clock.cc:648] HybridClock initialized: now 1781980796633420 us; error 38 us; skew 500 ppm
I20260620 18:39:56.634820 20931 webserver.cc:460] Webserver started at http://127.10.250.126:39671/ using document root <none> and password file <none>
I20260620 18:39:56.635020 20931 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:56.635061 20931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:56.635155 20931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:56.636279 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "ced5cd4c0bf7464d8086846f15610825"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "402c928606293005e6c80a7cf16cf6f7"
server_key_iv: "041d1174c830687c05d30a4af6dabb10"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.636601 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "ced5cd4c0bf7464d8086846f15610825"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "402c928606293005e6c80a7cf16cf6f7"
server_key_iv: "041d1174c830687c05d30a4af6dabb10"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.637830 20931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:56.638494 20945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.638715 20931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:56.638808 20931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "ced5cd4c0bf7464d8086846f15610825"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "402c928606293005e6c80a7cf16cf6f7"
server_key_iv: "041d1174c830687c05d30a4af6dabb10"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.638883 20931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:56.642153 20931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:56.642433 20931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:56.642596 20931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:56.646270 20931 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:37395
I20260620 18:39:56.646325 20997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:37395 every 8 connection(s)
I20260620 18:39:56.646732 20931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:56.647441 20998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:56.650215 20998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825: Bootstrap starting.
I20260620 18:39:56.650969 20998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:56.651286 20998 log.cc:824] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:56.652055 20998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825: No bootstrap required, opened a new log
I20260620 18:39:56.653646 20998 raft_consensus.cc:348] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced5cd4c0bf7464d8086846f15610825" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37395 } }
I20260620 18:39:56.653843 20998 raft_consensus.cc:374] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:56.653895 20998 raft_consensus.cc:729] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced5cd4c0bf7464d8086846f15610825, State: Initialized, Role: FOLLOWER
I20260620 18:39:56.654096 20998 consensus_queue.cc:260] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced5cd4c0bf7464d8086846f15610825" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37395 } }
I20260620 18:39:56.654201 20998 raft_consensus.cc:388] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:56.654244 20998 raft_consensus.cc:482] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:56.654295 20998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:56.654943 20998 raft_consensus.cc:504] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced5cd4c0bf7464d8086846f15610825" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37395 } }
I20260620 18:39:56.655068 20998 leader_election.cc:302] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ced5cd4c0bf7464d8086846f15610825; no voters:
I20260620 18:39:56.655238 20998 leader_election.cc:288] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:56.655288 21001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:56.655465 21001 raft_consensus.cc:686] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [term 1 LEADER]: Becoming Leader. State: Replica: ced5cd4c0bf7464d8086846f15610825, State: Running, Role: LEADER
I20260620 18:39:56.655504 20998 sys_catalog.cc:564] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:56.655575 21001 consensus_queue.cc:237] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced5cd4c0bf7464d8086846f15610825" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37395 } }
I20260620 18:39:56.655979 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 20931
I20260620 18:39:56.656075 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:56.656205 21003 sys_catalog.cc:455] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ced5cd4c0bf7464d8086846f15610825. Latest consensus state: current_term: 1 leader_uuid: "ced5cd4c0bf7464d8086846f15610825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced5cd4c0bf7464d8086846f15610825" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37395 } } }
I20260620 18:39:56.656275 21003 sys_catalog.cc:458] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:56.656288 11241 external_mini_cluster.cc:1428] Setting key 6a06b8ac2c031a2fcce22056db46dcdd
I20260620 18:39:56.656627 21002 sys_catalog.cc:455] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ced5cd4c0bf7464d8086846f15610825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced5cd4c0bf7464d8086846f15610825" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37395 } } }
I20260620 18:39:56.656731 21010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:56.656733 21002 sys_catalog.cc:458] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:56.657413 21010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:56.658938 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:56.656930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.10.250.126:37395)
0620 18:39:56.657106 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:56.657117 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:56.657218 (+ 101us) server_negotiation.cc:407] Connection header received
0620 18:39:56.657339 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:56.657345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:56.657380 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:56.657599 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:56.658320 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:56.658324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:56.658324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:56.658329 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:56.658371 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:56.658385 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:56.658453 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:56.658706 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 18:39:56.658768 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
I20260620 18:39:56.659076 21010 catalog_manager.cc:1347] Generated new cluster ID: 4a6f144c5bd84e08a238662a2158a9da
I20260620 18:39:56.659164 21010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:56.673345 21010 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:56.673568 21010 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:56.678643 21010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ced5cd4c0bf7464d8086846f15610825: Generated new TSK 0
I20260620 18:39:56.678910 21010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:39:56.684875 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37395
--builtin_ntp_servers=127.10.250.84:41017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:39:56.783463 21022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:56.783667 21022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:56.783691 21022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:56.784917 21022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:56.784961 21022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:56.784989 21022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:56.786386 21022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:56.786697 21022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:56.787019 21022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:56.793064 21028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:56.793121 21027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:56.793102 21030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:56.793262 21022 server_base.cc:1034] running on GCE node
I20260620 18:39:56.793642 21022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:56.793962 21022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:56.795142 21022 hybrid_clock.cc:648] HybridClock initialized: now 1781980796795104 us; error 47 us; skew 500 ppm
I20260620 18:39:56.796768 21022 webserver.cc:460] Webserver started at http://127.10.250.65:35269/ using document root <none> and password file <none>
I20260620 18:39:56.797037 21022 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:56.797171 21022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:56.797322 21022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:56.798758 21022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "2efbe4e924df43c593ed376cabd0f578"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "b0a21fec2f737a897bfd7ccebfb52e76"
server_key_iv: "72b4004bee2e4aa3e7ea238b0579eebb"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.799160 21022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "2efbe4e924df43c593ed376cabd0f578"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "b0a21fec2f737a897bfd7ccebfb52e76"
server_key_iv: "72b4004bee2e4aa3e7ea238b0579eebb"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.800715 21022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:39:56.801595 21036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.801824 21022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:56.801916 21022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "2efbe4e924df43c593ed376cabd0f578"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "b0a21fec2f737a897bfd7ccebfb52e76"
server_key_iv: "72b4004bee2e4aa3e7ea238b0579eebb"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.801990 21022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:56.806865 21022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:56.807233 21022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:56.807377 21022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:56.807608 21022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:56.807966 21022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:56.808010 21022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.808049 21022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:56.808074 21022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.815377 21022 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:41771
I20260620 18:39:56.815472 21149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:41771 every 8 connection(s)
I20260620 18:39:56.815841 21022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:56.819072 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:56.816627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:59607 (local address 127.10.250.126:37395)
0620 18:39:56.816691 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:56.816693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:56.816944 (+ 251us) server_negotiation.cc:407] Connection header received
0620 18:39:56.817405 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:56.817406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:56.817412 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:56.817495 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:56.818766 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:56.818767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:56.818768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:56.818774 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:56.818795 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:56.818802 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:56.818878 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:56.818970 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 18:39:56.818989 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:39:56.819298 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:56.816612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:37395 (local address 127.10.250.65:59607)
0620 18:39:56.816810 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:56.816852 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:56.817317 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:56.817556 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:56.817560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:56.817568 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:56.818584 (+ 1016us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:56.818585 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:56.818632 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:56.818663 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:56.818889 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:56.818893 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:56.818896 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:56.818953 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 18:39:56.819005 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260620 18:39:56.819823 21150 heartbeater.cc:347] Connected to a master server at 127.10.250.126:37395
I20260620 18:39:56.819921 21150 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:56.820101 21150 heartbeater.cc:511] Master 127.10.250.126:37395 requested a full tablet report, sending...
I20260620 18:39:56.820641 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 21022
I20260620 18:39:56.820654 20962 ts_manager.cc:194] Registered new tserver with Master: 2efbe4e924df43c593ed376cabd0f578 (127.10.250.65:41771)
I20260620 18:39:56.820735 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:56.820967 11241 external_mini_cluster.cc:1428] Setting key 9a8835c6055950a351d756e4959f045c
I20260620 18:39:56.821909 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37395
--builtin_ntp_servers=127.10.250.84:41017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:39:56.905066 21153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:56.905292 21153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:56.905314 21153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:56.906687 21153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:56.906733 21153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:56.906760 21153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:56.908179 21153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:56.908450 21153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:56.908766 21153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:56.914263 21158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:56.914292 21153 server_base.cc:1034] running on GCE node
W20260620 18:39:56.914208 21159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:56.914208 21161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:56.914683 21153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:56.915005 21153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:56.916173 21153 hybrid_clock.cc:648] HybridClock initialized: now 1781980796916149 us; error 40 us; skew 500 ppm
I20260620 18:39:56.917608 21153 webserver.cc:460] Webserver started at http://127.10.250.66:45907/ using document root <none> and password file <none>
I20260620 18:39:56.917898 21153 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:56.917977 21153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:56.918143 21153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:56.919369 21153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "d156c106191646e68b488cb2bea358a6"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "73c0a7118dc4fa213aa4fcfd0ab6eec6"
server_key_iv: "48aeae95d46a0dc26521cfd2c0e72ae8"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.919768 21153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "d156c106191646e68b488cb2bea358a6"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "73c0a7118dc4fa213aa4fcfd0ab6eec6"
server_key_iv: "48aeae95d46a0dc26521cfd2c0e72ae8"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.921200 21153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:39:56.922096 21167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.922290 21153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:56.922415 21153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "d156c106191646e68b488cb2bea358a6"
format_stamp: "Formatted at 2026-06-20 18:39:56 on dist-test-slave-wp6g"
server_key: "73c0a7118dc4fa213aa4fcfd0ab6eec6"
server_key_iv: "48aeae95d46a0dc26521cfd2c0e72ae8"
server_key_version: "encryptionkey@0"
I20260620 18:39:56.922539 21153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:56.926397 21153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:56.926753 21153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:56.926910 21153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:56.927150 21153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:56.927644 21153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:56.927698 21153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.927742 21153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:56.927769 21153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:56.934680 21153 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:35125
I20260620 18:39:56.934836 21280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:35125 every 8 connection(s)
I20260620 18:39:56.935160 21153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:56.936736 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 21153
I20260620 18:39:56.936842 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:56.937065 11241 external_mini_cluster.cc:1428] Setting key 59ea8d3ba7eed00b108ed6d7209cc4ec
I20260620 18:39:56.938284 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37395
--builtin_ntp_servers=127.10.250.84:41017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 18:39:56.938727 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:56.936000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:38167 (local address 127.10.250.126:37395)
0620 18:39:56.936117 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:56.936119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:56.936260 (+ 141us) server_negotiation.cc:407] Connection header received
0620 18:39:56.936742 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:56.936743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:56.936751 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:56.936839 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:56.938323 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:56.938324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:56.938325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:56.938331 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:56.938353 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:56.938359 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:56.938473 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:56.938653 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 18:39:56.938668 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 18:39:56.939011 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:56.935931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:37395 (local address 127.10.250.66:38167)
0620 18:39:56.936127 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:56.936169 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:56.936641 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:56.936931 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:56.936935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:56.936946 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:56.938138 (+ 1192us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:56.938138 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:56.938171 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:56.938203 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:56.938537 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:56.938550 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:56.938555 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:39:56.938647 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:39:56.938731 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260620 18:39:56.940153 21281 heartbeater.cc:347] Connected to a master server at 127.10.250.126:37395
I20260620 18:39:56.940277 21281 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:56.940466 21281 heartbeater.cc:511] Master 127.10.250.126:37395 requested a full tablet report, sending...
I20260620 18:39:56.940866 20962 ts_manager.cc:194] Registered new tserver with Master: d156c106191646e68b488cb2bea358a6 (127.10.250.66:35125)
W20260620 18:39:57.026093 21284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:57.026355 21284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:57.026396 21284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:57.027696 21284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:57.027767 21284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:57.027812 21284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:57.029224 21284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:57.029511 21284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:57.029817 21284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:57.034960 21290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:57.034960 21289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:57.035174 21284 server_base.cc:1034] running on GCE node
W20260620 18:39:57.035007 21292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:57.035463 21284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:57.035794 21284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:57.036971 21284 hybrid_clock.cc:648] HybridClock initialized: now 1781980797036962 us; error 40 us; skew 500 ppm
I20260620 18:39:57.038357 21284 webserver.cc:460] Webserver started at http://127.10.250.67:41589/ using document root <none> and password file <none>
I20260620 18:39:57.038704 21284 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:57.038779 21284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:57.038923 21284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:57.040138 21284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "4972f56f64b944c484e3db3e0b4933f6"
format_stamp: "Formatted at 2026-06-20 18:39:57 on dist-test-slave-wp6g"
server_key: "db183868762cb0a2c222c8c3e6a715a6"
server_key_iv: "e3f7a84b239f15a3dcb8b9334a535855"
server_key_version: "encryptionkey@0"
I20260620 18:39:57.040515 21284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "4972f56f64b944c484e3db3e0b4933f6"
format_stamp: "Formatted at 2026-06-20 18:39:57 on dist-test-slave-wp6g"
server_key: "db183868762cb0a2c222c8c3e6a715a6"
server_key_iv: "e3f7a84b239f15a3dcb8b9334a535855"
server_key_version: "encryptionkey@0"
I20260620 18:39:57.041999 21284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:57.042927 21298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:57.043104 21284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:57.043206 21284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "4972f56f64b944c484e3db3e0b4933f6"
format_stamp: "Formatted at 2026-06-20 18:39:57 on dist-test-slave-wp6g"
server_key: "db183868762cb0a2c222c8c3e6a715a6"
server_key_iv: "e3f7a84b239f15a3dcb8b9334a535855"
server_key_version: "encryptionkey@0"
I20260620 18:39:57.043298 21284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:57.046718 21284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:57.047027 21284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:57.047185 21284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:57.047396 21284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:57.047711 21284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:57.047765 21284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:57.047804 21284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:57.047832 21284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:57.054013 21284 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:33987
I20260620 18:39:57.054059 21411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:33987 every 8 connection(s)
I20260620 18:39:57.054412 21284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:57.058244 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:57.055233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:56531 (local address 127.10.250.126:37395)
0620 18:39:57.055301 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:57.055303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:57.055601 (+ 298us) server_negotiation.cc:407] Connection header received
0620 18:39:57.056503 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:57.056505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:57.056512 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:57.056555 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:57.057901 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:57.057902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:57.057903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:57.057907 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:57.057924 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:57.057929 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:57.058001 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:57.058150 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 18:39:57.058167 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 18:39:57.058458 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:57.055212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:37395 (local address 127.10.250.67:56531)
0620 18:39:57.055418 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:57.055452 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:57.056364 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:57.056641 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:57.056655 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:57.056665 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:57.057725 (+ 1060us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:57.057725 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:57.057756 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:57.057789 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:57.058054 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:57.058059 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:57.058062 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:57.058160 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:39:57.058227 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260620 18:39:57.058975 21412 heartbeater.cc:347] Connected to a master server at 127.10.250.126:37395
I20260620 18:39:57.059068 21412 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:57.059185 21412 heartbeater.cc:511] Master 127.10.250.126:37395 requested a full tablet report, sending...
I20260620 18:39:57.059525 20962 ts_manager.cc:194] Registered new tserver with Master: 4972f56f64b944c484e3db3e0b4933f6 (127.10.250.67:33987)
I20260620 18:39:57.063997 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 21284
I20260620 18:39:57.064111 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:57.064352 11241 external_mini_cluster.cc:1428] Setting key f13212425c069a88e808e2e9cc8d3f8c
I20260620 18:39:57.065443 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:39:57.067507 21019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:57.067002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.10.250.126:37395)
0620 18:39:57.067103 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:57.067105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:57.067136 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:39:57.067226 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:57.067227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:57.067233 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:57.067315 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:57.067445 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36838: BlockingRecv error: recv got EOF from 127.0.0.1:36838 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:39:57.067734 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 21022
I20260620 18:39:57.073283 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 21153
I20260620 18:39:57.077795 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 21284
I20260620 18:39:57.082653 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 20931
2026-06-20T18:39:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (568 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:57 dist-test-slave-wp6g krb5kdc[21422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:57 dist-test-slave-wp6g krb5kdc[21422](info): set up 2 sockets
Jun 20 18:39:57 dist-test-slave-wp6g krb5kdc[21422](info): commencing operation
krb5kdc: starting...
W20260620 18:39:58.127641 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:39:58 dist-test-slave-wp6g krb5kdc[21422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:39:58.253502 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:33315
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:33315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:58.336356 21435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:58.336578 21435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:58.336603 21435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:58.337798 21435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:58.337842 21435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:58.337857 21435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:58.337870 21435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:58.337882 21435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:58.339531 21435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:33315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:33315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:58.339798 21435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:58.340065 21435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:58.345220 21440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:58.345224 21441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:58.345279 21443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:58.345512 21435 server_base.cc:1034] running on GCE node
I20260620 18:39:58.345808 21435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:58.346180 21435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:58.347357 21435 hybrid_clock.cc:648] HybridClock initialized: now 1781980798347332 us; error 40 us; skew 500 ppm
Jun 20 18:39:58 dist-test-slave-wp6g krb5kdc[21422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980798, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:58.351195 21435 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:39:58.351760 21435 webserver.cc:460] Webserver started at http://127.10.250.126:44121/ using document root <none> and password file <none>
I20260620 18:39:58.352005 21435 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:58.352075 21435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:58.352238 21435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:58.353376 21435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "8560451cbc1d42c99afdf37362011d2d"
format_stamp: "Formatted at 2026-06-20 18:39:58 on dist-test-slave-wp6g"
server_key: "9b2e2663a8ed1488152b202478a27aa5"
server_key_iv: "3d17f9be818aeb8d67113ba7f1c3c5db"
server_key_version: "encryptionkey@0"
I20260620 18:39:58.353790 21435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "8560451cbc1d42c99afdf37362011d2d"
format_stamp: "Formatted at 2026-06-20 18:39:58 on dist-test-slave-wp6g"
server_key: "9b2e2663a8ed1488152b202478a27aa5"
server_key_iv: "3d17f9be818aeb8d67113ba7f1c3c5db"
server_key_version: "encryptionkey@0"
I20260620 18:39:58.355304 21435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:39:58.356143 21450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:58.356321 21435 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:58.356431 21435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "8560451cbc1d42c99afdf37362011d2d"
format_stamp: "Formatted at 2026-06-20 18:39:58 on dist-test-slave-wp6g"
server_key: "9b2e2663a8ed1488152b202478a27aa5"
server_key_iv: "3d17f9be818aeb8d67113ba7f1c3c5db"
server_key_version: "encryptionkey@0"
I20260620 18:39:58.356509 21435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:58.368678 21435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:58.370947 21435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:58.371129 21435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:58.375464 21435 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:33315
I20260620 18:39:58.375505 21502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:33315 every 8 connection(s)
I20260620 18:39:58.375896 21435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:58.377267 21503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:58.378595 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 21435
I20260620 18:39:58.378718 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:39:58.378898 11241 external_mini_cluster.cc:1428] Setting key b1040c4982c73ea23f010a0e5288508f
I20260620 18:39:58.380369 21503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d: Bootstrap starting.
I20260620 18:39:58.381253 21503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:58.381574 21503 log.cc:824] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:58.382411 21503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d: No bootstrap required, opened a new log
I20260620 18:39:58.384810 21503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8560451cbc1d42c99afdf37362011d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 33315 } }
I20260620 18:39:58.384977 21503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:58.385001 21503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8560451cbc1d42c99afdf37362011d2d, State: Initialized, Role: FOLLOWER
I20260620 18:39:58.385162 21503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8560451cbc1d42c99afdf37362011d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 33315 } }
I20260620 18:39:58.385260 21503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:58.385298 21503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:58.385351 21503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:58.385953 21503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8560451cbc1d42c99afdf37362011d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 33315 } }
I20260620 18:39:58.386087 21503 leader_election.cc:302] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8560451cbc1d42c99afdf37362011d2d; no voters:
I20260620 18:39:58.386327 21503 leader_election.cc:288] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:58.386356 21508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:58.386484 21508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [term 1 LEADER]: Becoming Leader. State: Replica: 8560451cbc1d42c99afdf37362011d2d, State: Running, Role: LEADER
Jun 20 18:39:58 dist-test-slave-wp6g krb5kdc[21422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:58.386641 21503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:58.386695 21508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8560451cbc1d42c99afdf37362011d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 33315 } }
I20260620 18:39:58.387023 21508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8560451cbc1d42c99afdf37362011d2d. Latest consensus state: current_term: 1 leader_uuid: "8560451cbc1d42c99afdf37362011d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8560451cbc1d42c99afdf37362011d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 33315 } } }
I20260620 18:39:58.387194 21508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:58.387794 21510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8560451cbc1d42c99afdf37362011d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8560451cbc1d42c99afdf37362011d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 33315 } } }
I20260620 18:39:58.387939 21510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d [sys.catalog]: This master's current role is: LEADER
W20260620 18:39:58.388757 21524 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:39:58.388813 21524 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:39:58.388856 21517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:58.388993 21517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:58.390980 21517 catalog_manager.cc:1347] Generated new cluster ID: 0247cc1ee7ad4f72a2911459b8497168
I20260620 18:39:58.391052 21517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:58.391178 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:58.379439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.10.250.126:33315)
0620 18:39:58.379641 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:58.379651 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:58.379687 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:39:58.380235 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:58.380245 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:58.380309 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:58.380481 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:58.381160 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.382579 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.383410 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.384078 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.384224 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:58.387226 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:58.387230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:58.387231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:58.387245 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:58.389346 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:58.389749 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:58.389752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:58.389753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:58.389870 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:58.390091 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:58.390091 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:58.390092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:58.390413 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:58.390593 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:58.390772 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 18:39:58.390885 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260620 18:39:58.404223 21517 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:58.404803 21517 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:58.408018 21517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d: Generated new TSK 0
I20260620 18:39:58.408274 21517 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:39:58.445199 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:33315
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:58.527792 21529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:58.527998 21529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:58.528021 21529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:58.529174 21529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:58.529218 21529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:58.529245 21529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:39:58.530786 21529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:33315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:58.531016 21529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:58.531255 21529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:58.536626 21537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:58.536661 21534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:58.536635 21535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:58.536746 21529 server_base.cc:1034] running on GCE node
I20260620 18:39:58.537039 21529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:58.537271 21529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:58.538412 21529 hybrid_clock.cc:648] HybridClock initialized: now 1781980798538401 us; error 28 us; skew 500 ppm
Jun 20 18:39:58 dist-test-slave-wp6g krb5kdc[21422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980798, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:58.541260 21529 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:39:58.541733 21529 webserver.cc:460] Webserver started at http://127.10.250.65:40175/ using document root <none> and password file <none>
I20260620 18:39:58.542014 21529 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:58.542062 21529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:58.542199 21529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:58.543336 21529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "76d2b90d7dd944f8bc64ed969082d170"
format_stamp: "Formatted at 2026-06-20 18:39:58 on dist-test-slave-wp6g"
server_key: "f15e9f12cfda6e8398aa9196a9fceea9"
server_key_iv: "284089ca846bc2afcd99efeeecac13ab"
server_key_version: "encryptionkey@0"
I20260620 18:39:58.543677 21529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "76d2b90d7dd944f8bc64ed969082d170"
format_stamp: "Formatted at 2026-06-20 18:39:58 on dist-test-slave-wp6g"
server_key: "f15e9f12cfda6e8398aa9196a9fceea9"
server_key_iv: "284089ca846bc2afcd99efeeecac13ab"
server_key_version: "encryptionkey@0"
I20260620 18:39:58.544983 21529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:39:58.545749 21544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:58.545960 21529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:39:58.546048 21529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "76d2b90d7dd944f8bc64ed969082d170"
format_stamp: "Formatted at 2026-06-20 18:39:58 on dist-test-slave-wp6g"
server_key: "f15e9f12cfda6e8398aa9196a9fceea9"
server_key_iv: "284089ca846bc2afcd99efeeecac13ab"
server_key_version: "encryptionkey@0"
I20260620 18:39:58.546125 21529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:58.579207 21529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:58.581467 21529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:58.581629 21529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:58.581879 21529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:58.582214 21529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:58.582268 21529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:58.582305 21529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:58.582333 21529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:58.589017 21529 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:38445
I20260620 18:39:58.589062 21657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:38445 every 8 connection(s)
I20260620 18:39:58.589416 21529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:58.590642 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 21529
I20260620 18:39:58.590744 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:58.590924 11241 external_mini_cluster.cc:1428] Setting key db74b538e5f044a9b280bbbc83d6c483
Jun 20 18:39:58 dist-test-slave-wp6g krb5kdc[21422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980798, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
I20260620 18:39:58.600893 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:58.590289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:37201 (local address 127.10.250.126:33315)
0620 18:39:58.590371 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:58.590373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:58.590658 (+ 285us) server_negotiation.cc:407] Connection header received
0620 18:39:58.591347 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:58.591348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:58.591359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:58.591417 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:58.592779 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.593365 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.594862 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.595229 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.595307 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:58.597583 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:58.597585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:58.597585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:58.597594 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:58.599449 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:58.599912 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:58.599913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:58.599914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:58.599979 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:58.600231 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:58.600232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:58.600232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:58.600350 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:58.600404 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:58.600679 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 18:39:58.600756 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:39:58.601146 21660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:58.590246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:33315 (local address 127.10.250.65:37201)
0620 18:39:58.590467 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:58.590543 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:58.591254 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:58.591476 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:58.591479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:58.592359 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:58.592668 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:58.592676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.593501 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.593503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:58.594777 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:58.594781 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.595320 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.595321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:58.595433 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:58.596068 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:58.596082 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:58.597472 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:58.599543 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:58.599550 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:58.599552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:58.599825 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:58.600053 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:58.600054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:58.600054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:58.600170 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:58.600411 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:58.600416 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:58.600550 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 18:39:58.600649 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:39:58.600848 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":97,"threads_started":1}
I20260620 18:39:58.601701 21658 heartbeater.cc:347] Connected to a master server at 127.10.250.126:33315
I20260620 18:39:58.601812 21658 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:58.602020 21658 heartbeater.cc:511] Master 127.10.250.126:33315 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
I20260620 18:39:58.602655 21467 ts_manager.cc:194] Registered new tserver with Master: 76d2b90d7dd944f8bc64ed969082d170 (127.10.250.65:38445)
I20260620 18:39:58.603332 21467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:37201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:39:58.617956 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:33315
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:58.700963 21663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:58.701238 21663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:58.701277 21663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:58.702559 21663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:58.702646 21663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:58.702692 21663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:39:58.704252 21663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:33315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:58.704552 21663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:58.704805 21663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:58.710289 21668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:58.710263 21669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:58.710461 21671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:58.710740 21663 server_base.cc:1034] running on GCE node
I20260620 18:39:58.710963 21663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:58.711239 21663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:58.712407 21663 hybrid_clock.cc:648] HybridClock initialized: now 1781980798712353 us; error 69 us; skew 500 ppm
Jun 20 18:39:58 dist-test-slave-wp6g krb5kdc[21422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980798, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:58.715260 21663 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:39:58.715745 21663 webserver.cc:460] Webserver started at http://127.10.250.66:39021/ using document root <none> and password file <none>
I20260620 18:39:58.716009 21663 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:58.716078 21663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:58.716229 21663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:58.717325 21663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "f4c608c0dc964bd3a5165f2720f31c60"
format_stamp: "Formatted at 2026-06-20 18:39:58 on dist-test-slave-wp6g"
server_key: "728d228dcfaa3ad32bff618d3db953db"
server_key_iv: "1eb7525812b3b39d50a3ea9670c37d31"
server_key_version: "encryptionkey@0"
I20260620 18:39:58.717697 21663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "f4c608c0dc964bd3a5165f2720f31c60"
format_stamp: "Formatted at 2026-06-20 18:39:58 on dist-test-slave-wp6g"
server_key: "728d228dcfaa3ad32bff618d3db953db"
server_key_iv: "1eb7525812b3b39d50a3ea9670c37d31"
server_key_version: "encryptionkey@0"
I20260620 18:39:58.719224 21663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:39:58.720125 21678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:58.720389 21663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:39:58.720535 21663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "f4c608c0dc964bd3a5165f2720f31c60"
format_stamp: "Formatted at 2026-06-20 18:39:58 on dist-test-slave-wp6g"
server_key: "728d228dcfaa3ad32bff618d3db953db"
server_key_iv: "1eb7525812b3b39d50a3ea9670c37d31"
server_key_version: "encryptionkey@0"
I20260620 18:39:58.720624 21663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:58.749496 21663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:58.751888 21663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:58.752064 21663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:58.752359 21663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:58.752702 21663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:58.752758 21663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:58.752802 21663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:58.752830 21663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:58.759142 21663 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:38897
I20260620 18:39:58.759191 21791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:38897 every 8 connection(s)
I20260620 18:39:58.759539 21663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:58.763386 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 21663
I20260620 18:39:58.763504 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:58.763684 11241 external_mini_cluster.cc:1428] Setting key 58a708a7e58010f901d54ba7179379f1
Jun 20 18:39:58 dist-test-slave-wp6g krb5kdc[21422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980798, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:58.769419 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:58.760312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:36341 (local address 127.10.250.126:33315)
0620 18:39:58.760373 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:58.760375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:58.760612 (+ 237us) server_negotiation.cc:407] Connection header received
0620 18:39:58.761175 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:58.761177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:58.761190 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:58.761241 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:58.762197 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.762798 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.763867 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.764100 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.764167 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:58.766410 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:58.766411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:58.766412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:58.766420 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:58.767992 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:58.768415 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:58.768417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:58.768417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:58.768480 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:58.768751 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:58.768752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:58.768752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:58.768877 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:58.768948 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:58.769241 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 18:39:58.769322 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:39:58.769642 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:58.760316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:33315 (local address 127.10.250.66:36341)
0620 18:39:58.760467 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:58.760505 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:58.761058 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:58.761314 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:58.761318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:58.761902 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:58.762095 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:58.762100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.762888 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.762889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:58.763775 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:58.763777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.764181 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.764182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:58.764346 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:58.764931 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:58.764939 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:58.766308 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:58.768094 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:58.768101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:58.768104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:58.768338 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:58.768549 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:58.768550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:58.768550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:58.768677 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:58.768949 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:58.768957 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:58.769108 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 18:39:58.769220 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:39:58.769387 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260620 18:39:58.770037 21792 heartbeater.cc:347] Connected to a master server at 127.10.250.126:33315
I20260620 18:39:58.770143 21792 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:58.770345 21792 heartbeater.cc:511] Master 127.10.250.126:33315 requested a full tablet report, sending...
I20260620 18:39:58.770735 21467 ts_manager.cc:194] Registered new tserver with Master: f4c608c0dc964bd3a5165f2720f31c60 (127.10.250.66:38897)
I20260620 18:39:58.771318 21467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:36341
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:39:58.793995 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:33315
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:58.876734 21797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:58.877018 21797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:58.877059 21797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:58.878350 21797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:58.878446 21797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:58.878497 21797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:39:58.880141 21797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:33315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:39:58.880461 21797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:58.880715 21797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:58.885977 21802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:58.885985 21803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:58.886031 21805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:58.886245 21797 server_base.cc:1034] running on GCE node
I20260620 18:39:58.886492 21797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:58.886852 21797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:58.888041 21797 hybrid_clock.cc:648] HybridClock initialized: now 1781980798888017 us; error 41 us; skew 500 ppm
Jun 20 18:39:58 dist-test-slave-wp6g krb5kdc[21422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980798, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:58.890913 21797 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:39:58.891464 21797 webserver.cc:460] Webserver started at http://127.10.250.67:44091/ using document root <none> and password file <none>
I20260620 18:39:58.891732 21797 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:58.891809 21797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:58.891983 21797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:58.893177 21797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "9491fba96ae742b8991c51f8b3a6472c"
format_stamp: "Formatted at 2026-06-20 18:39:58 on dist-test-slave-wp6g"
server_key: "c8977de2ac9c925e03c18db3f638d36b"
server_key_iv: "43068af2be66db73e42cea6180de33bd"
server_key_version: "encryptionkey@0"
I20260620 18:39:58.893599 21797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "9491fba96ae742b8991c51f8b3a6472c"
format_stamp: "Formatted at 2026-06-20 18:39:58 on dist-test-slave-wp6g"
server_key: "c8977de2ac9c925e03c18db3f638d36b"
server_key_iv: "43068af2be66db73e42cea6180de33bd"
server_key_version: "encryptionkey@0"
I20260620 18:39:58.895217 21797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 18:39:58.896046 21812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:58.896284 21797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:39:58.896401 21797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "9491fba96ae742b8991c51f8b3a6472c"
format_stamp: "Formatted at 2026-06-20 18:39:58 on dist-test-slave-wp6g"
server_key: "c8977de2ac9c925e03c18db3f638d36b"
server_key_iv: "43068af2be66db73e42cea6180de33bd"
server_key_version: "encryptionkey@0"
I20260620 18:39:58.896479 21797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:58.927420 21797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:58.929659 21797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:58.929863 21797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:58.930124 21797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:58.930449 21797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:58.930521 21797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:58.930609 21797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:58.930639 21797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:58.936776 21797 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:36071
I20260620 18:39:58.936820 21925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:36071 every 8 connection(s)
I20260620 18:39:58.937152 21797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:58.939563 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 21797
I20260620 18:39:58.939643 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:58.939818 11241 external_mini_cluster.cc:1428] Setting key e2bd57c886b6b87429eba799dc12f941
Jun 20 18:39:58 dist-test-slave-wp6g krb5kdc[21422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980798, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:39:58.947324 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:58.937868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:45373 (local address 127.10.250.126:33315)
0620 18:39:58.937916 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:58.937918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:58.938125 (+ 207us) server_negotiation.cc:407] Connection header received
0620 18:39:58.938709 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:58.938711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:58.938722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:58.938788 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:58.940269 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.940948 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.942072 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.942258 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.942316 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:58.944555 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:58.944556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:58.944557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:58.944563 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:58.945985 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:58.946379 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:58.946380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:58.946380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:58.946430 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:58.946726 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:58.946727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:58.946727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:58.946831 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:58.946884 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:58.947195 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 18:39:58.947262 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:39:58.947446 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:58.937816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:33315 (local address 127.10.250.67:45373)
0620 18:39:58.937985 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:58.938030 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:58.938596 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:58.938862 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:58.938865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:58.939863 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:58.940169 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:58.940177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.941061 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.941062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:58.941952 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:58.941962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.942340 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.942341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:58.942458 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:58.943102 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:58.943110 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:58.944461 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:58.946095 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:58.946100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:58.946102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:58.946313 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:58.946493 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:58.946494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:58.946494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:58.946667 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:58.946890 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:58.946894 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:58.947007 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 18:39:58.947117 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 18:39:58.947239 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":117,"threads_started":1}
I20260620 18:39:58.947968 21926 heartbeater.cc:347] Connected to a master server at 127.10.250.126:33315
I20260620 18:39:58.948082 21926 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:58.948271 21926 heartbeater.cc:511] Master 127.10.250.126:33315 requested a full tablet report, sending...
I20260620 18:39:58.948686 21467 ts_manager.cc:194] Registered new tserver with Master: 9491fba96ae742b8991c51f8b3a6472c (127.10.250.67:36071)
I20260620 18:39:58.949059 21467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:45373
I20260620 18:39:58.951885 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:39:58.959545 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:58.953268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.10.250.126:33315)
0620 18:39:58.953329 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:58.953331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:58.953471 (+ 140us) server_negotiation.cc:407] Connection header received
0620 18:39:58.953574 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:58.953575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:58.953585 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:58.953630 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:58.954182 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.954739 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.955473 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.955668 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.955743 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:58.956873 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:58.956874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:58.956875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:58.956882 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:58.958217 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:58.958658 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:58.958660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:58.958660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:58.958739 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:58.958971 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:58.958972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:58.958973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:58.959130 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:58.959243 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:58.959355 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 18:39:58.959432 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:39:58.961508 21467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:58.963384 21467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:58.970314 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:58.966072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:36071 (local address 127.0.0.1:58452)
0620 18:39:58.966569 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:58.966580 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:58.966620 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:58.966945 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:58.966946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:58.966947 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:58.966989 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:58.966991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.968194 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.968195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:58.969165 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:58.969168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.970037 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.970038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:58.970099 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:58.970110 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:39:58.970164 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 18:39:58.970228 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":92,"threads_started":1}
I20260620 18:39:58.970328 21939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:58.966109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58452 (local address 127.10.250.67:36071)
0620 18:39:58.966741 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:58.966746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:58.966755 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:39:58.966791 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:58.966793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:58.966808 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:58.966878 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:58.967060 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.968096 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.969255 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.969951 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.970030 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:58.970097 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:58.970190 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 18:39:58.970238 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":71,"threads_started":1}
I20260620 18:39:58.970690 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:58.966376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55402 (local address 127.10.250.65:38445)
0620 18:39:58.966578 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:58.966584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:58.966650 (+ 66us) server_negotiation.cc:407] Connection header received
0620 18:39:58.966743 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:58.966745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:58.966759 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:58.966873 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:58.967111 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.967940 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.969458 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.970161 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.970226 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:58.970277 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:58.970573 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 18:39:58.970616 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":109,"threads_started":1}
I20260620 18:39:58.970484 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:58.966264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:38445 (local address 127.0.0.1:55402)
0620 18:39:58.966593 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:58.966604 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:58.966666 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:58.966958 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:58.966959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:58.966961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:58.967030 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:58.967032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.968044 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.968045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:58.969367 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:58.969369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.970238 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.970239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:58.970302 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:58.970305 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:39:58.970357 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 18:39:58.970399 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":71,"thread_start_us":73,"threads_started":1}
I20260620 18:39:58.971321 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:58.966042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:38897 (local address 127.0.0.1:39660)
0620 18:39:58.966305 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:58.966350 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:58.966426 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:58.966722 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:58.966725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:58.966727 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:58.966802 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:58.966804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.968132 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.968133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:58.969362 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:58.969365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.971147 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.971148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:58.971194 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:58.971198 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:39:58.971242 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 18:39:58.971273 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":81,"thread_start_us":64,"threads_started":1}
I20260620 18:39:58.971369 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:58.966123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.10.250.66:38897)
0620 18:39:58.966293 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:58.966299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:58.966408 (+ 109us) server_negotiation.cc:407] Connection header received
0620 18:39:58.966532 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:58.966534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:58.966552 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:58.966630 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:58.966887 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.968026 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.970313 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:58.971054 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:58.971139 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:58.971193 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:58.971254 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 18:39:58.971293 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260620 18:39:58.971524 21860 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfac42f490843eabd5da6b9a48507b0 (DEFAULT_TABLE table=test-table [id=3babcf5b3edb43a4b5a3a6caf3ddb7d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:58.971915 21860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfac42f490843eabd5da6b9a48507b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:58.972147 21726 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfac42f490843eabd5da6b9a48507b0 (DEFAULT_TABLE table=test-table [id=3babcf5b3edb43a4b5a3a6caf3ddb7d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:58.972405 21726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfac42f490843eabd5da6b9a48507b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:58.975308 21944 tablet_bootstrap.cc:492] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60: Bootstrap starting.
I20260620 18:39:58.975356 21592 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfac42f490843eabd5da6b9a48507b0 (DEFAULT_TABLE table=test-table [id=3babcf5b3edb43a4b5a3a6caf3ddb7d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:58.975492 21943 tablet_bootstrap.cc:492] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c: Bootstrap starting.
I20260620 18:39:58.975730 21592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfac42f490843eabd5da6b9a48507b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:58.976097 21944 tablet_bootstrap.cc:654] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:58.976104 21943 tablet_bootstrap.cc:654] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:58.976665 21943 log.cc:824] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:58.976675 21944 log.cc:824] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:58.977499 21944 tablet_bootstrap.cc:492] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60: No bootstrap required, opened a new log
I20260620 18:39:58.977609 21944 ts_tablet_manager.cc:1397] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:39:58.978243 21943 tablet_bootstrap.cc:492] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c: No bootstrap required, opened a new log
I20260620 18:39:58.978335 21943 ts_tablet_manager.cc:1397] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:39:58.979492 21947 tablet_bootstrap.cc:492] T 4bfac42f490843eabd5da6b9a48507b0 P 76d2b90d7dd944f8bc64ed969082d170: Bootstrap starting.
I20260620 18:39:58.980000 21944 raft_consensus.cc:348] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c608c0dc964bd3a5165f2720f31c60" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 38897 } } peers { permanent_uuid: "9491fba96ae742b8991c51f8b3a6472c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36071 } } peers { permanent_uuid: "76d2b90d7dd944f8bc64ed969082d170" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38445 } }
I20260620 18:39:58.980181 21944 raft_consensus.cc:374] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:58.980247 21944 raft_consensus.cc:729] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c608c0dc964bd3a5165f2720f31c60, State: Initialized, Role: FOLLOWER
I20260620 18:39:58.980149 21943 raft_consensus.cc:348] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c608c0dc964bd3a5165f2720f31c60" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 38897 } } peers { permanent_uuid: "9491fba96ae742b8991c51f8b3a6472c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36071 } } peers { permanent_uuid: "76d2b90d7dd944f8bc64ed969082d170" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38445 } }
I20260620 18:39:58.980280 21947 tablet_bootstrap.cc:654] T 4bfac42f490843eabd5da6b9a48507b0 P 76d2b90d7dd944f8bc64ed969082d170: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:58.980321 21943 raft_consensus.cc:374] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:58.980357 21943 raft_consensus.cc:729] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9491fba96ae742b8991c51f8b3a6472c, State: Initialized, Role: FOLLOWER
I20260620 18:39:58.980396 21944 consensus_queue.cc:260] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c608c0dc964bd3a5165f2720f31c60" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 38897 } } peers { permanent_uuid: "9491fba96ae742b8991c51f8b3a6472c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36071 } } peers { permanent_uuid: "76d2b90d7dd944f8bc64ed969082d170" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38445 } }
I20260620 18:39:58.980487 21943 consensus_queue.cc:260] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c608c0dc964bd3a5165f2720f31c60" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 38897 } } peers { permanent_uuid: "9491fba96ae742b8991c51f8b3a6472c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36071 } } peers { permanent_uuid: "76d2b90d7dd944f8bc64ed969082d170" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38445 } }
I20260620 18:39:58.980723 21944 ts_tablet_manager.cc:1428] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:58.980736 21792 heartbeater.cc:503] Master 127.10.250.126:33315 was elected leader, sending a full tablet report...
I20260620 18:39:58.980810 21943 ts_tablet_manager.cc:1428] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:39:58.980954 21926 heartbeater.cc:503] Master 127.10.250.126:33315 was elected leader, sending a full tablet report...
I20260620 18:39:58.980942 21947 log.cc:824] T 4bfac42f490843eabd5da6b9a48507b0 P 76d2b90d7dd944f8bc64ed969082d170: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:58.981968 21947 tablet_bootstrap.cc:492] T 4bfac42f490843eabd5da6b9a48507b0 P 76d2b90d7dd944f8bc64ed969082d170: No bootstrap required, opened a new log
I20260620 18:39:58.982048 21947 ts_tablet_manager.cc:1397] T 4bfac42f490843eabd5da6b9a48507b0 P 76d2b90d7dd944f8bc64ed969082d170: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:39:58.983587 21947 raft_consensus.cc:348] T 4bfac42f490843eabd5da6b9a48507b0 P 76d2b90d7dd944f8bc64ed969082d170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c608c0dc964bd3a5165f2720f31c60" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 38897 } } peers { permanent_uuid: "9491fba96ae742b8991c51f8b3a6472c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36071 } } peers { permanent_uuid: "76d2b90d7dd944f8bc64ed969082d170" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38445 } }
I20260620 18:39:58.983731 21947 raft_consensus.cc:374] T 4bfac42f490843eabd5da6b9a48507b0 P 76d2b90d7dd944f8bc64ed969082d170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:58.983752 21947 raft_consensus.cc:729] T 4bfac42f490843eabd5da6b9a48507b0 P 76d2b90d7dd944f8bc64ed969082d170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d2b90d7dd944f8bc64ed969082d170, State: Initialized, Role: FOLLOWER
I20260620 18:39:58.983873 21947 consensus_queue.cc:260] T 4bfac42f490843eabd5da6b9a48507b0 P 76d2b90d7dd944f8bc64ed969082d170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c608c0dc964bd3a5165f2720f31c60" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 38897 } } peers { permanent_uuid: "9491fba96ae742b8991c51f8b3a6472c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36071 } } peers { permanent_uuid: "76d2b90d7dd944f8bc64ed969082d170" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38445 } }
I20260620 18:39:58.984252 21658 heartbeater.cc:503] Master 127.10.250.126:33315 was elected leader, sending a full tablet report...
I20260620 18:39:58.984380 21947 ts_tablet_manager.cc:1428] T 4bfac42f490843eabd5da6b9a48507b0 P 76d2b90d7dd944f8bc64ed969082d170: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260620 18:39:59.010162 21793 tablet.cc:2369] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:39:59.090246 21659 tablet.cc:2369] T 4bfac42f490843eabd5da6b9a48507b0 P 76d2b90d7dd944f8bc64ed969082d170: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:39:59.112974 21949 raft_consensus.cc:482] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:59.113111 21949 raft_consensus.cc:504] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c608c0dc964bd3a5165f2720f31c60" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 38897 } } peers { permanent_uuid: "9491fba96ae742b8991c51f8b3a6472c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36071 } } peers { permanent_uuid: "76d2b90d7dd944f8bc64ed969082d170" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38445 } }
I20260620 18:39:59.113513 21949 leader_election.cc:288] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4c608c0dc964bd3a5165f2720f31c60 (127.10.250.66:38897), 76d2b90d7dd944f8bc64ed969082d170 (127.10.250.65:38445)
I20260620 18:39:59.117020 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:59.113899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:43235 (local address 127.10.250.66:38897)
0620 18:39:59.113952 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:59.113954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:59.113973 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:39:59.114008 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:59.114009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:59.114021 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:59.114103 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:59.114343 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:59.114949 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:59.116179 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:59.116641 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:59.116699 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:59.116706 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:59.116900 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 18:39:59.116943 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 18:39:59.117031 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:59.113762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:38897 (local address 127.10.250.67:43235)
0620 18:39:59.113825 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:59.113839 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:59.113890 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:59.114130 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:59.114131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:59.114133 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:59.114236 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:59.114238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:59.115037 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:59.115038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:59.116095 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:59.116098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:59.116732 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:59.116733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:59.116824 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:59.116828 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:39:59.116881 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 18:39:59.116933 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 18:39:59.117122 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:59.114063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:47527 (local address 127.10.250.65:38445)
0620 18:39:59.114119 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:59.114121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:59.114225 (+ 104us) server_negotiation.cc:407] Connection header received
0620 18:39:59.114255 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:59.114256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:59.114266 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:59.114322 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:59.114547 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:59.115039 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:59.116311 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:59.116733 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:59.116813 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:59.116822 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:59.117001 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 18:39:59.117051 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 18:39:59.117156 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:59.113928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:38445 (local address 127.10.250.67:47527)
0620 18:39:59.114135 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:59.114146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:59.114207 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:59.114351 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:59.114352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:59.114353 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:59.114429 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:59.114431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:59.115181 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:59.115182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:59.116219 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:59.116222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:59.116826 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:59.116827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:59.116918 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:59.116923 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:39:59.116980 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 18:39:59.117016 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":120,"threads_started":1}
I20260620 18:39:59.117437 21746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfac42f490843eabd5da6b9a48507b0" candidate_uuid: "9491fba96ae742b8991c51f8b3a6472c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c608c0dc964bd3a5165f2720f31c60" is_pre_election: true
I20260620 18:39:59.117434 21612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfac42f490843eabd5da6b9a48507b0" candidate_uuid: "9491fba96ae742b8991c51f8b3a6472c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d2b90d7dd944f8bc64ed969082d170" is_pre_election: true
I20260620 18:39:59.117601 21612 raft_consensus.cc:2454] T 4bfac42f490843eabd5da6b9a48507b0 P 76d2b90d7dd944f8bc64ed969082d170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9491fba96ae742b8991c51f8b3a6472c in term 0.
I20260620 18:39:59.117601 21746 raft_consensus.cc:2454] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9491fba96ae742b8991c51f8b3a6472c in term 0.
I20260620 18:39:59.117835 21813 leader_election.cc:302] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9491fba96ae742b8991c51f8b3a6472c, f4c608c0dc964bd3a5165f2720f31c60; no voters:
I20260620 18:39:59.118024 21949 raft_consensus.cc:2781] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:59.118129 21949 raft_consensus.cc:482] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:59.118167 21949 raft_consensus.cc:3037] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:59.118971 21949 raft_consensus.cc:504] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c608c0dc964bd3a5165f2720f31c60" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 38897 } } peers { permanent_uuid: "9491fba96ae742b8991c51f8b3a6472c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36071 } } peers { permanent_uuid: "76d2b90d7dd944f8bc64ed969082d170" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38445 } }
I20260620 18:39:59.119158 21949 leader_election.cc:288] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [CANDIDATE]: Term 1 election: Requested vote from peers f4c608c0dc964bd3a5165f2720f31c60 (127.10.250.66:38897), 76d2b90d7dd944f8bc64ed969082d170 (127.10.250.65:38445)
I20260620 18:39:59.119441 21746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfac42f490843eabd5da6b9a48507b0" candidate_uuid: "9491fba96ae742b8991c51f8b3a6472c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c608c0dc964bd3a5165f2720f31c60"
I20260620 18:39:59.119462 21612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfac42f490843eabd5da6b9a48507b0" candidate_uuid: "9491fba96ae742b8991c51f8b3a6472c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d2b90d7dd944f8bc64ed969082d170"
I20260620 18:39:59.119542 21612 raft_consensus.cc:3037] T 4bfac42f490843eabd5da6b9a48507b0 P 76d2b90d7dd944f8bc64ed969082d170 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:59.119542 21746 raft_consensus.cc:3037] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:59.120477 21746 raft_consensus.cc:2454] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9491fba96ae742b8991c51f8b3a6472c in term 1.
I20260620 18:39:59.120477 21612 raft_consensus.cc:2454] T 4bfac42f490843eabd5da6b9a48507b0 P 76d2b90d7dd944f8bc64ed969082d170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9491fba96ae742b8991c51f8b3a6472c in term 1.
I20260620 18:39:59.120671 21813 leader_election.cc:302] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9491fba96ae742b8991c51f8b3a6472c, f4c608c0dc964bd3a5165f2720f31c60; no voters:
I20260620 18:39:59.120783 21949 raft_consensus.cc:2781] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:59.120971 21949 raft_consensus.cc:686] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [term 1 LEADER]: Becoming Leader. State: Replica: 9491fba96ae742b8991c51f8b3a6472c, State: Running, Role: LEADER
I20260620 18:39:59.121084 21949 consensus_queue.cc:237] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c608c0dc964bd3a5165f2720f31c60" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 38897 } } peers { permanent_uuid: "9491fba96ae742b8991c51f8b3a6472c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36071 } } peers { permanent_uuid: "76d2b90d7dd944f8bc64ed969082d170" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38445 } }
I20260620 18:39:59.121961 21467 catalog_manager.cc:5495] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9491fba96ae742b8991c51f8b3a6472c (127.10.250.67). New cstate: current_term: 1 leader_uuid: "9491fba96ae742b8991c51f8b3a6472c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c608c0dc964bd3a5165f2720f31c60" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 38897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9491fba96ae742b8991c51f8b3a6472c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76d2b90d7dd944f8bc64ed969082d170" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38445 } health_report { overall_health: UNKNOWN } } }
I20260620 18:39:59.156507 21939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:59.154062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.10.250.67:36071)
0620 18:39:59.154105 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:59.154107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:59.154132 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:39:59.154206 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:59.154207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:59.154226 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:59.154300 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:59.154406 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:59.154888 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:59.155700 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:59.155895 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:59.155991 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:59.156131 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:59.156236 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:59.156292 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:59.156341 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 18:39:59.156356 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:39:59.158660 21746 raft_consensus.cc:1261] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60 [term 1 FOLLOWER]: Refusing update from remote peer 9491fba96ae742b8991c51f8b3a6472c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:59.158660 21612 raft_consensus.cc:1261] T 4bfac42f490843eabd5da6b9a48507b0 P 76d2b90d7dd944f8bc64ed969082d170 [term 1 FOLLOWER]: Refusing update from remote peer 9491fba96ae742b8991c51f8b3a6472c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:59.159054 21949 consensus_queue.cc:1035] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d2b90d7dd944f8bc64ed969082d170" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:59.159220 21953 consensus_queue.cc:1035] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c608c0dc964bd3a5165f2720f31c60" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 38897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:59.161895 21957 mvcc.cc:205] Tried to move back new op lower bound from 7298993353350172672 to 7298993353200717824. Current Snapshot: MvccSnapshot[applied={T|T < 7298993353350172672}]
I20260620 18:39:59.161984 21958 mvcc.cc:205] Tried to move back new op lower bound from 7298993353350172672 to 7298993353200717824. Current Snapshot: MvccSnapshot[applied={T|T < 7298993353350172672}]
I20260620 18:39:59.164294 21467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:59.164412 21467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:59.165190 21467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8560451cbc1d42c99afdf37362011d2d: Sending DeleteTablet for 3 replicas of tablet 4bfac42f490843eabd5da6b9a48507b0
I20260620 18:39:59.165525 21860 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfac42f490843eabd5da6b9a48507b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:59 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:58452
I20260620 18:39:59.165567 21592 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfac42f490843eabd5da6b9a48507b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:59 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:55402
I20260620 18:39:59.165768 21966 tablet_replica.cc:330] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c: stopping tablet replica
I20260620 18:39:59.165763 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 21529
I20260620 18:39:59.165757 21726 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfac42f490843eabd5da6b9a48507b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:59 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:39660
I20260620 18:39:59.165863 21966 raft_consensus.cc:2229] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:59.165992 21966 raft_consensus.cc:2258] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:59.166644 21966 ts_tablet_manager.cc:1905] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:59.167948 21968 tablet_replica.cc:330] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60: stopping tablet replica
I20260620 18:39:59.168119 21968 raft_consensus.cc:2229] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:59.168282 21968 raft_consensus.cc:2258] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:59.168678 21966 ts_tablet_manager.cc:1918] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:59.168802 21966 log.cc:1196] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/4bfac42f490843eabd5da6b9a48507b0
I20260620 18:39:59.168821 21968 ts_tablet_manager.cc:1905] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:59.169193 21966 ts_tablet_manager.cc:1939] T 4bfac42f490843eabd5da6b9a48507b0 P 9491fba96ae742b8991c51f8b3a6472c: Deleting consensus metadata
I20260620 18:39:59.169678 21452 catalog_manager.cc:4841] TS 9491fba96ae742b8991c51f8b3a6472c (127.10.250.67:36071): tablet 4bfac42f490843eabd5da6b9a48507b0 (table test-table [id=3babcf5b3edb43a4b5a3a6caf3ddb7d7]) successfully deleted
I20260620 18:39:59.170745 21968 ts_tablet_manager.cc:1918] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:59.170840 21968 log.cc:1196] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/4bfac42f490843eabd5da6b9a48507b0
I20260620 18:39:59.171140 21968 ts_tablet_manager.cc:1939] T 4bfac42f490843eabd5da6b9a48507b0 P f4c608c0dc964bd3a5165f2720f31c60: Deleting consensus metadata
I20260620 18:39:59.171576 21451 catalog_manager.cc:4841] TS f4c608c0dc964bd3a5165f2720f31c60 (127.10.250.66:38897): tablet 4bfac42f490843eabd5da6b9a48507b0 (table test-table [id=3babcf5b3edb43a4b5a3a6caf3ddb7d7]) successfully deleted
W20260620 18:39:59.173520 21451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:38445 (error 108)
I20260620 18:39:59.173657 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 21663
I20260620 18:39:59.174794 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:59.174669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:38445 (local address 127.0.0.1:55404)
0620 18:39:59.174717 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:59.174757 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:38445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260620 18:39:59.174949 21451 catalog_manager.cc:4584] TS 76d2b90d7dd944f8bc64ed969082d170 (127.10.250.65:38445): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4bfac42f490843eabd5da6b9a48507b0: Network error: Client connection negotiation failed: client connection to 127.10.250.65:38445: connect: Connection refused (error 111)
I20260620 18:39:59.181237 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 21797
I20260620 18:39:59.185770 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 21435
2026-06-20T18:39:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2103 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:59 dist-test-slave-wp6g krb5kdc[21972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:59 dist-test-slave-wp6g krb5kdc[21972](info): set up 2 sockets
Jun 20 18:39:59 dist-test-slave-wp6g krb5kdc[21972](info): commencing operation
krb5kdc: starting...
W20260620 18:40:00.239647 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:40:00 dist-test-slave-wp6g krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.250.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.250.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.250.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/rangeradmin_127.10.250.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.250.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/rangeradmin_127.10.250.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.250.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.250.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.250.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/rangerlookup_127.10.250.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.250.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/rangerlookup_127.10.250.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.250.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.250.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.250.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/HTTP_127.10.250.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.250.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/HTTP_127.10.250.84@KRBTEST.COM.keytab.
I20260620 18:40:00.413538 11241 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGJBaTq/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/postgres -l logfile start
2026-06-20 18:40:03.282 UTC [22002] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 18:40:03.282 UTC [22002] LOG: listening on IPv4 address "127.10.250.84", port 38355
2026-06-20 18:40:03.286 UTC [22002] LOG: listening on Unix socket "/tmp/.s.PGSQL.38355"
2026-06-20 18:40:03.314 UTC [22005] LOG: database system was shut down at 2026-06-20 18:40:01 UTC
2026-06-20 18:40:03.317 UTC [22002] LOG: database system is ready to accept connections
I20260620 18:40:04.279410 11241 mini_postgres.cc:96] Postgres bound to 38355
2026-06-20 18:40:04.285 UTC [22013] FATAL: database "slave" does not exist
127.10.250.84:38355 - accepting connections
I20260620 18:40:04.286319 11241 mini_ranger.cc:161] Starting Ranger...
I20260620 18:40:04.301286 11241 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 18:40:04.838238 11241 mini_ranger.cc:87] Created ranger Postgres database
I20260620 18:40:04.838326 11241 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-admin
2026-06-20 18:40:05,119 [I] DB FLAVOR :POSTGRES
2026-06-20 18:40:05,119 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:40:05,119 [I] Checking connection..
2026-06-20 18:40:05,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 18:40:05,401 [I] Checking connection passed.
2026-06-20 18:40:05,401 [I] --------- Verifying version history table ---------
2026-06-20 18:40:05,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:40:05,653 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:40:05,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:40:05,909 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:40:05,910 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 18:40:05,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 18:40:06.180 UTC [22077] WARNING: there is no transaction in progress
2026-06-20 18:40:06,186 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 18:40:06,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:40:06,428 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 18:40:06,428 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:40:06,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 18:40:06,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 18:40:06,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wp6g.c.gcp-upstream.internal','N') ;"
2026-06-20 18:40:07,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:40:07,426 [I] Table x_portal_user does not exist in database ranger
2026-06-20 18:40:07,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:40:07,700 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 18:40:07,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:40:07,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:40:08,225 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 18:40:08,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 18:40:09.281 UTC [22185] WARNING: there is no transaction in progress
2026-06-20 18:40:09.299 UTC [22185] WARNING: there is no transaction in progress
2026-06-20 18:40:09.316 UTC [22185] WARNING: there is no transaction in progress
2026-06-20 18:40:09.333 UTC [22185] WARNING: there is no transaction in progress
2026-06-20 18:40:09.350 UTC [22185] WARNING: there is no transaction in progress
2026-06-20 18:40:09.368 UTC [22185] WARNING: there is no transaction in progress
2026-06-20 18:40:09.439 UTC [22185] WARNING: there is no transaction in progress
2026-06-20 18:40:09.447 UTC [22185] WARNING: there is no transaction in progress
2026-06-20 18:40:09.467 UTC [22185] WARNING: there is no transaction in progress
2026-06-20 18:40:09.480 UTC [22185] WARNING: there is no transaction in progress
2026-06-20 18:40:09.495 UTC [22185] WARNING: there is no transaction in progress
2026-06-20 18:40:09.514 UTC [22185] WARNING: there is no transaction in progress
2026-06-20 18:40:09.526 UTC [22185] WARNING: there is no transaction in progress
2026-06-20 18:40:09.534 UTC [22185] WARNING: there is no transaction in progress
2026-06-20 18:40:09.544 UTC [22185] WARNING: there is no transaction in progress
2026-06-20 18:40:09.552 UTC [22185] WARNING: there is no transaction in progress
2026-06-20 18:40:10,355 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 18:40:10,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 18:40:10,624 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 18:40:10,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:40:10,884 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 18:40:10,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:40:11,131 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 18:40:11,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:40:11,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:40:11,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wp6g.c.gcp-upstream.internal';"
2026-06-20 18:40:11,910 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 18:40:11,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 18:40:12,171 [I] DB_PATCHES have already been applied
I20260620 18:40:12.177273 11241 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGJBaTq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 18:40:12.177362 11241 mini_ranger.cc:193] Using host: 127.10.250.84
I20260620 18:40:12.180858 11241 mini_ranger.cc:227] Ranger admin URL: http://127.10.250.84:53849
Jun 20, 2026 6:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:40:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:40:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 18:40:13.298594 11241 mini_ranger.cc:160] Time spent starting Ranger: real 9.012s user 0.005s sys 0.005s
Jun 20, 2026 6:40:33 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:40:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 6:40:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:40:33 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:40:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:40:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:40:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:40:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:40:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:40:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:40:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:40:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:40:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:40:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:40:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:40:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:40:33 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:40:34 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 18:40:34.952749 11241 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.250.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.250.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.250.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/rangerkms_127.10.250.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.250.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/rangerkms_127.10.250.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.250.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.250.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.250.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/HTTP_127.10.250.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.250.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/HTTP_127.10.250.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 18:40:35 dist-test-slave-wp6g krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980835, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 18:40:35.051741 11241 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 18:40:35.062664 11241 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 18:40:36.052371 11241 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 18:40:36.052500 11241 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-kms
I20260620 18:40:36.052522 11241 mini_ranger_kms.cc:227] Using postgres at 127.10.250.84:38355
2026-06-20 18:40:36,231 [I] DB FLAVOR :POSTGRES
2026-06-20 18:40:36,231 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:40:36,231 [I] Checking connection
2026-06-20 18:40:36,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 18:40:36,491 [I] connection success
2026-06-20 18:40:36,491 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 18:40:36,491 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 18:40:36,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 18:40:36,778 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 18:40:36,778 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:40:36,778 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 18:40:36,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:38355/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 18:40:37,060 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 18:40:37.571101 11241 mini_ranger_kms.cc:310] Created kms service
I20260620 18:40:37.775692 11241 mini_ranger_kms.cc:326] Created kudu user
I20260620 18:40:37.824867 11241 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 18:40:37 dist-test-slave-wp6g krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980835, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
I20260620 18:40:38.250361 11241 mini_ranger_kms.cc:384] Added ranger policy
I20260620 18:40:38.250600 11241 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 18:40:38.250629 11241 mini_ranger_kms.cc:240] Using host: 127.10.250.84
I20260620 18:40:38.253985 11241 mini_ranger_kms.cc:276] Ranger KMS PID: 22365
I20260620 18:40:38.254117 11241 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.250.84:42053
Jun 20, 2026 6:40:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:40:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:40:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:40:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:40:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:40:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 18:40:40.531543 11241 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.480s user 0.007s sys 0.152s
I20260620 18:40:40.531661 11241 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 18:40:40.531739 11241 mini_ranger_kms.cc:401] 127.10.250.84:42053/kms/v1/keys
Jun 20 18:40:40 dist-test-slave-wp6g krb5kdc[21972](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980840, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.250.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 18:40:41 dist-test-slave-wp6g krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980840, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.250.84@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
Jun 20 18:40:41 dist-test-slave-wp6g krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980840, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.250.84@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
Jun 20 18:40:43 dist-test-slave-wp6g krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:40:43.910229 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:35453
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:40313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:35453
--ranger_config_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.250.84:42053/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:43.995601 22416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:43.995818 22416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:43.995842 22416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:43.997026 22416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:43.997071 22416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:43.997085 22416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:43.997097 22416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:43.997110 22416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:43.998808 22416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:40313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.250.84:42053/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:35453
--ranger_config_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:35453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:40:43.999081 22416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:43.999330 22416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:44.004580 22421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:44.004580 22422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:44.004595 22424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:44.005004 22416 server_base.cc:1034] running on GCE node
I20260620 18:40:44.005367 22416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:44.005709 22416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:44.006886 22416 hybrid_clock.cc:648] HybridClock initialized: now 1781980844006858 us; error 42 us; skew 500 ppm
Jun 20 18:40:44 dist-test-slave-wp6g krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980844, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:44.009785 22416 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:40:44.010358 22416 webserver.cc:460] Webserver started at http://127.10.250.126:37165/ using document root <none> and password file <none>
I20260620 18:40:44.010676 22416 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:44.010726 22416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:44.010885 22416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:40:44 dist-test-slave-wp6g krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980844, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
I20260620 18:40:44.071817 22416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "0e2f9b1f4baa4646b75d5c09c8811c9c"
format_stamp: "Formatted at 2026-06-20 18:40:44 on dist-test-slave-wp6g"
server_key: "04f5148127063dad5e40e07bb538adf7"
server_key_iv: "35f973c3f69e9f2a7fd1e3b50d106630"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:44.072326 22416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "0e2f9b1f4baa4646b75d5c09c8811c9c"
format_stamp: "Formatted at 2026-06-20 18:40:44 on dist-test-slave-wp6g"
server_key: "04f5148127063dad5e40e07bb538adf7"
server_key_iv: "35f973c3f69e9f2a7fd1e3b50d106630"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:44.074232 22416 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 18:40:44.089916 22431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:44.090164 22416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:40:44.090273 22416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "0e2f9b1f4baa4646b75d5c09c8811c9c"
format_stamp: "Formatted at 2026-06-20 18:40:44 on dist-test-slave-wp6g"
server_key: "04f5148127063dad5e40e07bb538adf7"
server_key_iv: "35f973c3f69e9f2a7fd1e3b50d106630"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:44.090375 22416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:44.111375 22416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:44.113790 22416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:44.113972 22416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:44.119447 22416 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:35453
I20260620 18:40:44.119493 22483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:35453 every 8 connection(s)
I20260620 18:40:44.119961 22416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:44.120690 22484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:44.123116 22484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c: Bootstrap starting.
I20260620 18:40:44.123723 22484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:44.124034 22484 log.cc:824] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:44.124720 22484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c: No bootstrap required, opened a new log
I20260620 18:40:44.126832 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 22416
I20260620 18:40:44.126933 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:40:44.127128 11241 external_mini_cluster.cc:1428] Setting key 2edf3eab0d2c1787746aca519f1287dd
I20260620 18:40:44.127147 22484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2f9b1f4baa4646b75d5c09c8811c9c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 35453 } }
I20260620 18:40:44.127419 22484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:44.127488 22484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2f9b1f4baa4646b75d5c09c8811c9c, State: Initialized, Role: FOLLOWER
I20260620 18:40:44.127678 22484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2f9b1f4baa4646b75d5c09c8811c9c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 35453 } }
I20260620 18:40:44.127777 22484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:44.127815 22484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:44.127890 22484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:44.128583 22484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2f9b1f4baa4646b75d5c09c8811c9c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 35453 } }
I20260620 18:40:44.128710 22484 leader_election.cc:302] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e2f9b1f4baa4646b75d5c09c8811c9c; no voters:
I20260620 18:40:44.128918 22484 leader_election.cc:288] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:44.128969 22489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:44.129191 22489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [term 1 LEADER]: Becoming Leader. State: Replica: 0e2f9b1f4baa4646b75d5c09c8811c9c, State: Running, Role: LEADER
I20260620 18:40:44.129292 22484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:44.130574 22489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2f9b1f4baa4646b75d5c09c8811c9c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 35453 } }
I20260620 18:40:44.132591 22491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e2f9b1f4baa4646b75d5c09c8811c9c. Latest consensus state: current_term: 1 leader_uuid: "0e2f9b1f4baa4646b75d5c09c8811c9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2f9b1f4baa4646b75d5c09c8811c9c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 35453 } } }
I20260620 18:40:44.132685 22491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:44.132902 22503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:44.133275 22484 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:40:44.133420 22503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 18:40:44 dist-test-slave-wp6g krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:40:44.137070 22490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e2f9b1f4baa4646b75d5c09c8811c9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2f9b1f4baa4646b75d5c09c8811c9c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 35453 } } }
I20260620 18:40:44.137338 22490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:44.141755 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:44.127680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.10.250.126:35453)
0620 18:40:44.127888 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:44.127898 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:44.128112 (+ 214us) server_negotiation.cc:407] Connection header received
0620 18:40:44.128245 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:44.128253 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:44.128312 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:44.128468 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:44.129245 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:44.130735 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:44.131922 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:44.132541 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:44.132667 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:44.136160 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:44.136205 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:44.136206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:44.136303 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:44.139183 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:44.139632 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:44.139635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:44.139636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:44.139776 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:44.140011 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:44.140012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:44.140013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:44.140431 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:44.140569 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:44.141055 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 18:40:44.141231 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":90,"threads_started":1}
I20260620 18:40:44.147086 22503 catalog_manager.cc:1347] Generated new cluster ID: cc73954edcf8490596a4027a46a1ade2
I20260620 18:40:44.147156 22503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:44.175447 22503 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:44.176373 22503 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:44.183322 22503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c: Generated new TSK 0
I20260620 18:40:44.183665 22503 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:40:45.033305 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.250.84:42053/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:35453
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:40313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:45.168955 22529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:45.169281 22529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:45.169334 22529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:45.171178 22529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:45.171257 22529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:45.171306 22529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:40:45.173748 22529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:40313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.250.84:42053/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:35453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:40:45.174067 22529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:45.174414 22529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:45.180472 22537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:45.180480 22534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:45.181051 22535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:45.181464 22529 server_base.cc:1034] running on GCE node
I20260620 18:40:45.181721 22529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:45.182055 22529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:45.183231 22529 hybrid_clock.cc:648] HybridClock initialized: now 1781980845183211 us; error 39 us; skew 500 ppm
Jun 20 18:40:45 dist-test-slave-wp6g krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980845, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:45.186501 22529 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:40:45.187060 22529 webserver.cc:460] Webserver started at http://127.10.250.65:37747/ using document root <none> and password file <none>
I20260620 18:40:45.187307 22529 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:45.187376 22529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:45.187523 22529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:40:45 dist-test-slave-wp6g krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980845, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
I20260620 18:40:45.200356 22529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "ebfe220f6eb74261832c72711de5103a"
format_stamp: "Formatted at 2026-06-20 18:40:45 on dist-test-slave-wp6g"
server_key: "719373b6725315a9fe18633e44ef1a63"
server_key_iv: "ab003e444063d782920fbbeb6a556595"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:45.200966 22529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "ebfe220f6eb74261832c72711de5103a"
format_stamp: "Formatted at 2026-06-20 18:40:45 on dist-test-slave-wp6g"
server_key: "719373b6725315a9fe18633e44ef1a63"
server_key_iv: "ab003e444063d782920fbbeb6a556595"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:45.203604 22529 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 18:40:45.222823 22544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:45.223063 22529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:40:45.223170 22529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "ebfe220f6eb74261832c72711de5103a"
format_stamp: "Formatted at 2026-06-20 18:40:45 on dist-test-slave-wp6g"
server_key: "719373b6725315a9fe18633e44ef1a63"
server_key_iv: "ab003e444063d782920fbbeb6a556595"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:45.223294 22529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:45.237149 22529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:45.239836 22529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:45.240080 22529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:45.240368 22529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:45.240792 22529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:45.240871 22529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:45.240929 22529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:45.240974 22529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:45.247465 22529 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:38469
I20260620 18:40:45.247793 22659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:38469 every 8 connection(s)
I20260620 18:40:45.247990 22529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:40:45.256237 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 22529
I20260620 18:40:45.256347 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:40:45.256525 11241 external_mini_cluster.cc:1428] Setting key 5bb9599c58793f83d43249146ec53049
Jun 20 18:40:45 dist-test-slave-wp6g krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980845, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:40:45.269071 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.248789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:35453 (local address 127.10.250.65:56043)
0620 18:40:45.248981 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:45.249038 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:45.249725 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:45.259217 (+ 9492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:45.259229 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:45.259494 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:45.259794 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:45.259801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.260817 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.260819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:45.262116 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:45.262119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.262713 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.262714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:45.262857 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.263525 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:45.263536 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:45.265247 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:45.267392 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:45.267401 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:45.267404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:45.267712 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:45.267928 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:45.267929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:45.267929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:45.268064 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:45.268363 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:45.268369 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:45.268545 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 18:40:45.268657 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:40:45.268825 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
I20260620 18:40:45.269670 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.249035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:56043 (local address 127.10.250.126:35453)
0620 18:40:45.258940 (+ 9905us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:45.258948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:45.258958 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:40:45.258981 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:45.258982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:45.258994 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:45.259087 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:45.259909 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.260696 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.262234 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.262564 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.262630 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.265362 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:45.265363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:45.265363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:45.265372 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:45.267289 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:45.267799 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:45.267800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:45.267801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:45.267865 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:45.268128 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:45.268129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:45.268129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:45.268281 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:45.268329 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:45.269470 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0620 18:40:45.269561 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9875,"thread_start_us":53,"threads_started":1}
I20260620 18:40:45.270364 22660 heartbeater.cc:347] Connected to a master server at 127.10.250.126:35453
I20260620 18:40:45.270488 22660 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:45.270771 22660 heartbeater.cc:511] Master 127.10.250.126:35453 requested a full tablet report, sending...
I20260620 18:40:45.271459 22448 ts_manager.cc:194] Registered new tserver with Master: ebfe220f6eb74261832c72711de5103a (127.10.250.65:38469)
I20260620 18:40:45.272349 22448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:56043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:40:45.293530 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.250.84:42053/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:35453
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:40313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:45.410288 22666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:45.410598 22666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:45.410640 22666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:45.412468 22666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:45.412575 22666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:45.412648 22666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:40:45.414990 22666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:40313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.250.84:42053/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:35453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:40:45.415411 22666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:45.415714 22666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 20 18:40:45 dist-test-slave-wp6g krb5kdc[21972](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980845, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 18:40:45.422264 22674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:45.422722 22673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:45.423045 22676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:45.423808 22666 server_base.cc:1034] running on GCE node
I20260620 18:40:45.424096 22666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:45.424422 22666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:45.425616 22666 hybrid_clock.cc:648] HybridClock initialized: now 1781980845425590 us; error 39 us; skew 500 ppm
Jun 20 18:40:45 dist-test-slave-wp6g krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980845, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:45.428973 22666 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:40:45.429548 22666 webserver.cc:460] Webserver started at http://127.10.250.66:33063/ using document root <none> and password file <none>
I20260620 18:40:45.429874 22666 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:45.429956 22666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:45.430106 22666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:40:45 dist-test-slave-wp6g krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980845, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
I20260620 18:40:45.445147 22666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "9793277e296e4f6792df9fc1c46b0d00"
format_stamp: "Formatted at 2026-06-20 18:40:45 on dist-test-slave-wp6g"
server_key: "c30869ee7a093afe422067b9aeeb0b7f"
server_key_iv: "83bb5c27cab177f5005f92e447c1322f"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:45.445700 22666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "9793277e296e4f6792df9fc1c46b0d00"
format_stamp: "Formatted at 2026-06-20 18:40:45 on dist-test-slave-wp6g"
server_key: "c30869ee7a093afe422067b9aeeb0b7f"
server_key_iv: "83bb5c27cab177f5005f92e447c1322f"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:45.467162 22666 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.002s sys 0.000s
I20260620 18:40:45.482683 22683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:45.482957 22666 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260620 18:40:45.483052 22666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "9793277e296e4f6792df9fc1c46b0d00"
format_stamp: "Formatted at 2026-06-20 18:40:45 on dist-test-slave-wp6g"
server_key: "c30869ee7a093afe422067b9aeeb0b7f"
server_key_iv: "83bb5c27cab177f5005f92e447c1322f"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:45.483145 22666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:45.510183 22666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:45.512900 22666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:45.513064 22666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:45.513322 22666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:45.513710 22666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:45.513762 22666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:45.513797 22666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:45.513867 22666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:45.520742 22666 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:44133
I20260620 18:40:45.521298 22666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:40:45.522718 22797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:44133 every 8 connection(s)
I20260620 18:40:45.530347 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 22666
I20260620 18:40:45.530491 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:40:45.530736 11241 external_mini_cluster.cc:1428] Setting key e92243c4502310d4680a4d9384c12155
Jun 20 18:40:45 dist-test-slave-wp6g krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980845, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:40:45.539202 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.524054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:45377 (local address 127.10.250.126:35453)
0620 18:40:45.524101 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:45.524103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:45.524362 (+ 259us) server_negotiation.cc:407] Connection header received
0620 18:40:45.525066 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:45.525067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:45.525084 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:45.525115 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:45.526001 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.526840 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.532379 (+ 5539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.532730 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.532778 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.535697 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:45.535698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:45.535699 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:45.535708 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:45.537396 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:45.537964 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:45.537965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:45.537966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:45.538030 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:45.538352 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:45.538353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:45.538353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:45.538495 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:45.538566 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:45.538982 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 18:40:45.539087 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:40:45.540052 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.523770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:35453 (local address 127.10.250.66:45377)
0620 18:40:45.524229 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:45.524277 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:45.524932 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:45.525332 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:45.525337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:45.525589 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:45.525883 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:45.525890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.527013 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.527014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:45.532255 (+ 5241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:45.532261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.532908 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.532909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:45.533109 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.533772 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:45.533783 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:45.535561 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:45.537505 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:45.537514 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:45.537517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:45.537859 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:45.538105 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:45.538106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:45.538106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:45.538275 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:45.538615 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:45.538621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:45.538810 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 18:40:45.539605 (+ 795us) client_negotiation.cc:240] Negotiation successful
0620 18:40:45.539799 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":79,"threads_started":1}
I20260620 18:40:45.540894 22798 heartbeater.cc:347] Connected to a master server at 127.10.250.126:35453
I20260620 18:40:45.540999 22798 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:45.541219 22798 heartbeater.cc:511] Master 127.10.250.126:35453 requested a full tablet report, sending...
I20260620 18:40:45.541638 22448 ts_manager.cc:194] Registered new tserver with Master: 9793277e296e4f6792df9fc1c46b0d00 (127.10.250.66:44133)
I20260620 18:40:45.542148 22448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:45377
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:40:45.577287 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.250.84:42053/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:35453
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:40313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:45.706882 22803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:45.707342 22803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:45.707438 22803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:45.709246 22803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:45.709501 22803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:45.709591 22803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:40:45.711942 22803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:40313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.250.84:42053/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:35453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:40:45.712484 22803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:45.712837 22803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:45.719312 22811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:45.719316 22809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:45.720652 22803 server_base.cc:1034] running on GCE node
W20260620 18:40:45.720758 22808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:45.721032 22803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:45.721359 22803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:45.722563 22803 hybrid_clock.cc:648] HybridClock initialized: now 1781980845722529 us; error 40 us; skew 500 ppm
Jun 20 18:40:45 dist-test-slave-wp6g krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980845, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:45.726142 22803 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:40:45.726759 22803 webserver.cc:460] Webserver started at http://127.10.250.67:40401/ using document root <none> and password file <none>
I20260620 18:40:45.727070 22803 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:45.727246 22803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:45.727443 22803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:40:45 dist-test-slave-wp6g krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980845, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
I20260620 18:40:45.746961 22803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "75a67649472f4ae599412823668debb5"
format_stamp: "Formatted at 2026-06-20 18:40:45 on dist-test-slave-wp6g"
server_key: "fd8529f0f61f57aad46d0010c6d85881"
server_key_iv: "b5ae60d5335287e0fc2ce5e436a9b483"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:45.747529 22803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "75a67649472f4ae599412823668debb5"
format_stamp: "Formatted at 2026-06-20 18:40:45 on dist-test-slave-wp6g"
server_key: "fd8529f0f61f57aad46d0010c6d85881"
server_key_iv: "b5ae60d5335287e0fc2ce5e436a9b483"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:45.750387 22803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:40:45.765455 22818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:45.765717 22803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:40:45.765837 22803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "75a67649472f4ae599412823668debb5"
format_stamp: "Formatted at 2026-06-20 18:40:45 on dist-test-slave-wp6g"
server_key: "fd8529f0f61f57aad46d0010c6d85881"
server_key_iv: "b5ae60d5335287e0fc2ce5e436a9b483"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:45.765991 22803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:45.782550 22803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:45.785342 22803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:45.785544 22803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:45.792665 22803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:45.793210 22803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:45.793292 22803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:45.793354 22803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:45.793398 22803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:45.800587 22803 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:46373
I20260620 18:40:45.801080 22803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:40:45.801538 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 22803
I20260620 18:40:45.801636 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:40:45.801841 11241 external_mini_cluster.cc:1428] Setting key d7af03dadc357d80fe472a3aecf272ab
I20260620 18:40:45.803548 22931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:46373 every 8 connection(s)
Jun 20 18:40:45 dist-test-slave-wp6g krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980845, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:40:45.824457 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.803509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:38393 (local address 127.10.250.126:35453)
0620 18:40:45.803561 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:45.803564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:45.804474 (+ 910us) server_negotiation.cc:407] Connection header received
0620 18:40:45.805121 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:45.805122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:45.805133 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:45.805162 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:45.806007 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.806847 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.808363 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.808667 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.808708 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.820811 (+ 12103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:45.820813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:45.820813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:45.820823 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:45.822657 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:45.823280 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:45.823281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:45.823282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:45.823344 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:45.823656 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:45.823656 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:45.823657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:45.823806 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:45.823849 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:45.824263 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 18:40:45.824353 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 18:40:45.825817 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.803246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:35453 (local address 127.10.250.67:38393)
0620 18:40:45.804342 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:45.804392 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:45.805003 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:45.805367 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:45.805372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:45.805612 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:45.805899 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:45.805906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.806965 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.806966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:45.808263 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:45.808266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.808797 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.808798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:45.808940 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.818717 (+ 9777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:45.818732 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:45.820655 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:45.822800 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:45.822813 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:45.822817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:45.823157 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:45.823416 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:45.823417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:45.823417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:45.823575 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:45.823896 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:45.823904 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:45.824092 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 18:40:45.824851 (+ 759us) client_negotiation.cc:240] Negotiation successful
0620 18:40:45.825325 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"thread_start_us":78,"threads_started":1}
I20260620 18:40:45.826478 22932 heartbeater.cc:347] Connected to a master server at 127.10.250.126:35453
I20260620 18:40:45.826606 22932 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:45.826799 22932 heartbeater.cc:511] Master 127.10.250.126:35453 requested a full tablet report, sending...
I20260620 18:40:45.827201 22448 ts_manager.cc:194] Registered new tserver with Master: 75a67649472f4ae599412823668debb5 (127.10.250.67:46373)
I20260620 18:40:45.827715 22448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:38393
I20260620 18:40:45.836360 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:40:45.856612 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.838947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.10.250.126:35453)
0620 18:40:45.838995 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:45.838997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:45.839290 (+ 293us) server_negotiation.cc:407] Connection header received
0620 18:40:45.839395 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:45.839396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:45.839408 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:45.839439 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:45.844804 (+ 5365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.845593 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.847552 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.847843 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.847895 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.850126 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:45.850128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:45.850128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:45.850137 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:45.852264 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:45.855591 (+ 3327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:45.855593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:45.855593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:45.855673 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:45.855913 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:45.855915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:45.855915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:45.856064 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:45.856174 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:45.856412 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 18:40:45.856509 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:40:45.859552 22448 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:40:45.862032 22448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:40:45.877418 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.866247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:44133 (local address 127.0.0.1:53296)
0620 18:40:45.870562 (+ 4315us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:45.870620 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:45.870710 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:45.873677 (+ 2967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:45.873678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:45.873680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:45.873732 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:45.873733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.874960 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.874961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:45.876029 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:45.876031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.877090 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.877091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:45.877177 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.877194 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 18:40:45.877260 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 18:40:45.877321 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4251,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260620 18:40:45.878825 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.866481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:38469 (local address 127.0.0.1:56140)
0620 18:40:45.870847 (+ 4366us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:45.870856 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:45.870904 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:45.873238 (+ 2334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:45.873242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:45.873245 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:45.873327 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:45.873329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.874677 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.874678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:45.876028 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:45.876031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.878580 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.878581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:45.878648 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.878653 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:40:45.878699 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 18:40:45.878746 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4352,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260620 18:40:45.879266 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.866881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56140 (local address 127.10.250.65:38469)
0620 18:40:45.872994 (+ 6113us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:45.872999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:45.873008 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:40:45.873041 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:45.873043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:45.873059 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:45.873132 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:45.873420 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.874583 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.876462 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.877175 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.877252 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.877310 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:45.879126 (+ 1816us) server_negotiation.cc:299] Negotiation successful
0620 18:40:45.879179 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6078,"thread_start_us":54,"threads_started":1}
I20260620 18:40:45.879443 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.866638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53296 (local address 127.10.250.66:44133)
0620 18:40:45.872656 (+ 6018us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:45.872662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:45.872675 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:40:45.872723 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:45.872725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:45.872744 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:45.872855 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:45.873822 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.874873 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.876158 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.876916 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.876985 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.877053 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:45.879304 (+ 2251us) server_negotiation.cc:299] Negotiation successful
0620 18:40:45.879362 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5973,"thread_start_us":86,"threads_started":1}
I20260620 18:40:45.880199 22594 tablet_service.cc:1467] Processing CreateTablet for tablet 0600142f922541cface782b00b97c0c0 (DEFAULT_TABLE table=test-table [id=5b89bc45f6994a679ab765b2849f9a6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:45.880184 22732 tablet_service.cc:1467] Processing CreateTablet for tablet 0600142f922541cface782b00b97c0c0 (DEFAULT_TABLE table=test-table [id=5b89bc45f6994a679ab765b2849f9a6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:45.880560 22732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0600142f922541cface782b00b97c0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:45.880560 22594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0600142f922541cface782b00b97c0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:45.885710 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.866451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41096 (local address 127.10.250.67:46373)
0620 18:40:45.876217 (+ 9766us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:45.876222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:45.876233 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:40:45.876273 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:45.876275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:45.876293 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:45.876379 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:45.877352 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.878460 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.884415 (+ 5955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.885268 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.885364 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.885439 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:45.885574 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 18:40:45.885617 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9711,"thread_start_us":104,"threads_started":1}
I20260620 18:40:45.885707 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.866247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:46373 (local address 127.0.0.1:41096)
0620 18:40:45.870652 (+ 4405us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:45.870672 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:45.870731 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:45.876384 (+ 5653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:45.876385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:45.876388 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:45.876435 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:45.876436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.878583 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.878584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:45.879840 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:45.879844 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.885371 (+ 5527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.885372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:45.885458 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.885464 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:40:45.885559 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:40:45.885617 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4383,"mutex_wait_us":2,"thread_start_us":77,"threads_started":1}
I20260620 18:40:45.886713 22866 tablet_service.cc:1467] Processing CreateTablet for tablet 0600142f922541cface782b00b97c0c0 (DEFAULT_TABLE table=test-table [id=5b89bc45f6994a679ab765b2849f9a6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:45.887089 22866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0600142f922541cface782b00b97c0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:45.890825 22950 tablet_bootstrap.cc:492] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a: Bootstrap starting.
I20260620 18:40:45.891065 22951 tablet_bootstrap.cc:492] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00: Bootstrap starting.
I20260620 18:40:45.892109 22951 tablet_bootstrap.cc:654] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:45.892910 22950 tablet_bootstrap.cc:654] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:45.893291 22950 log.cc:824] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:45.893687 22951 log.cc:824] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:45.894287 22952 tablet_bootstrap.cc:492] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5: Bootstrap starting.
I20260620 18:40:45.895470 22952 tablet_bootstrap.cc:654] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:45.895797 22952 log.cc:824] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:45.896626 22951 tablet_bootstrap.cc:492] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00: No bootstrap required, opened a new log
I20260620 18:40:45.896740 22951 ts_tablet_manager.cc:1397] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 18:40:45.898947 22951 raft_consensus.cc:348] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793277e296e4f6792df9fc1c46b0d00" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44133 } } peers { permanent_uuid: "75a67649472f4ae599412823668debb5" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 46373 } } peers { permanent_uuid: "ebfe220f6eb74261832c72711de5103a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38469 } }
I20260620 18:40:45.899281 22951 raft_consensus.cc:374] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:45.899369 22951 raft_consensus.cc:729] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793277e296e4f6792df9fc1c46b0d00, State: Initialized, Role: FOLLOWER
I20260620 18:40:45.899570 22951 consensus_queue.cc:260] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793277e296e4f6792df9fc1c46b0d00" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44133 } } peers { permanent_uuid: "75a67649472f4ae599412823668debb5" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 46373 } } peers { permanent_uuid: "ebfe220f6eb74261832c72711de5103a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38469 } }
I20260620 18:40:45.899942 22951 ts_tablet_manager.cc:1428] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:40:45.900095 22798 heartbeater.cc:503] Master 127.10.250.126:35453 was elected leader, sending a full tablet report...
I20260620 18:40:45.901731 22952 tablet_bootstrap.cc:492] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5: No bootstrap required, opened a new log
I20260620 18:40:45.901854 22952 ts_tablet_manager.cc:1397] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260620 18:40:45.903842 22952 raft_consensus.cc:348] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793277e296e4f6792df9fc1c46b0d00" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44133 } } peers { permanent_uuid: "75a67649472f4ae599412823668debb5" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 46373 } } peers { permanent_uuid: "ebfe220f6eb74261832c72711de5103a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38469 } }
I20260620 18:40:45.904069 22952 raft_consensus.cc:374] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:45.904150 22952 raft_consensus.cc:729] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a67649472f4ae599412823668debb5, State: Initialized, Role: FOLLOWER
I20260620 18:40:45.904361 22952 consensus_queue.cc:260] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793277e296e4f6792df9fc1c46b0d00" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44133 } } peers { permanent_uuid: "75a67649472f4ae599412823668debb5" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 46373 } } peers { permanent_uuid: "ebfe220f6eb74261832c72711de5103a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38469 } }
I20260620 18:40:45.904982 22952 ts_tablet_manager.cc:1428] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:40:45.905136 22932 heartbeater.cc:503] Master 127.10.250.126:35453 was elected leader, sending a full tablet report...
I20260620 18:40:45.906944 22950 tablet_bootstrap.cc:492] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a: No bootstrap required, opened a new log
I20260620 18:40:45.907019 22950 ts_tablet_manager.cc:1397] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260620 18:40:45.909415 22950 raft_consensus.cc:348] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793277e296e4f6792df9fc1c46b0d00" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44133 } } peers { permanent_uuid: "75a67649472f4ae599412823668debb5" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 46373 } } peers { permanent_uuid: "ebfe220f6eb74261832c72711de5103a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38469 } }
I20260620 18:40:45.909579 22950 raft_consensus.cc:374] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:45.909607 22950 raft_consensus.cc:729] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebfe220f6eb74261832c72711de5103a, State: Initialized, Role: FOLLOWER
I20260620 18:40:45.909864 22950 consensus_queue.cc:260] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793277e296e4f6792df9fc1c46b0d00" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44133 } } peers { permanent_uuid: "75a67649472f4ae599412823668debb5" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 46373 } } peers { permanent_uuid: "ebfe220f6eb74261832c72711de5103a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38469 } }
I20260620 18:40:45.910192 22950 ts_tablet_manager.cc:1428] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:40:45.910341 22660 heartbeater.cc:503] Master 127.10.250.126:35453 was elected leader, sending a full tablet report...
I20260620 18:40:45.935310 22958 raft_consensus.cc:482] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:40:45.935573 22958 raft_consensus.cc:504] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793277e296e4f6792df9fc1c46b0d00" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44133 } } peers { permanent_uuid: "75a67649472f4ae599412823668debb5" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 46373 } } peers { permanent_uuid: "ebfe220f6eb74261832c72711de5103a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38469 } }
I20260620 18:40:45.936857 22958 leader_election.cc:288] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9793277e296e4f6792df9fc1c46b0d00 (127.10.250.66:44133), 75a67649472f4ae599412823668debb5 (127.10.250.67:46373)
I20260620 18:40:45.950356 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.936446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:56665 (local address 127.10.250.66:44133)
0620 18:40:45.936491 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:45.936493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:45.937381 (+ 888us) server_negotiation.cc:407] Connection header received
0620 18:40:45.937485 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:45.937486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:45.937498 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:45.937526 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:45.937764 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.938496 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.948701 (+ 10205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.949277 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.949332 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.949342 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:45.950231 (+ 889us) server_negotiation.cc:299] Negotiation successful
0620 18:40:45.950277 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:40:45.950874 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.936239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:44133 (local address 127.10.250.65:56665)
0620 18:40:45.937307 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:45.937321 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:45.937412 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:45.937594 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:45.937595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:45.937596 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:45.937696 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:45.937698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.938763 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.938764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:45.948574 (+ 9810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:45.948578 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.950064 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.950065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:45.950143 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.950147 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:40:45.950731 (+ 584us) client_negotiation.cc:240] Negotiation successful
0620 18:40:45.950787 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"thread_start_us":105,"threads_started":1}
I20260620 18:40:45.951359 22752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0600142f922541cface782b00b97c0c0" candidate_uuid: "ebfe220f6eb74261832c72711de5103a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793277e296e4f6792df9fc1c46b0d00" is_pre_election: true
I20260620 18:40:45.951532 22752 raft_consensus.cc:2454] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebfe220f6eb74261832c72711de5103a in term 0.
I20260620 18:40:45.951785 22549 leader_election.cc:302] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793277e296e4f6792df9fc1c46b0d00, ebfe220f6eb74261832c72711de5103a; no voters:
I20260620 18:40:45.952024 22958 raft_consensus.cc:2781] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:40:45.952135 22958 raft_consensus.cc:482] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:40:45.952196 22958 raft_consensus.cc:3037] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:45.953166 22958 raft_consensus.cc:504] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793277e296e4f6792df9fc1c46b0d00" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44133 } } peers { permanent_uuid: "75a67649472f4ae599412823668debb5" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 46373 } } peers { permanent_uuid: "ebfe220f6eb74261832c72711de5103a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38469 } }
I20260620 18:40:45.953500 22752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0600142f922541cface782b00b97c0c0" candidate_uuid: "ebfe220f6eb74261832c72711de5103a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793277e296e4f6792df9fc1c46b0d00"
I20260620 18:40:45.953573 22752 raft_consensus.cc:3037] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:45.954432 22752 raft_consensus.cc:2454] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebfe220f6eb74261832c72711de5103a in term 1.
I20260620 18:40:45.956887 22549 leader_election.cc:302] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793277e296e4f6792df9fc1c46b0d00, ebfe220f6eb74261832c72711de5103a; no voters:
I20260620 18:40:45.957207 22958 leader_election.cc:288] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [CANDIDATE]: Term 1 election: Requested vote from peers 9793277e296e4f6792df9fc1c46b0d00 (127.10.250.66:44133), 75a67649472f4ae599412823668debb5 (127.10.250.67:46373)
I20260620 18:40:45.957289 22958 raft_consensus.cc:2781] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:45.957365 22958 raft_consensus.cc:686] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [term 1 LEADER]: Becoming Leader. State: Replica: ebfe220f6eb74261832c72711de5103a, State: Running, Role: LEADER
I20260620 18:40:45.957499 22958 consensus_queue.cc:237] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793277e296e4f6792df9fc1c46b0d00" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44133 } } peers { permanent_uuid: "75a67649472f4ae599412823668debb5" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 46373 } } peers { permanent_uuid: "ebfe220f6eb74261832c72711de5103a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38469 } }
I20260620 18:40:45.958464 22448 catalog_manager.cc:5495] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a reported cstate change: term changed from 0 to 1, leader changed from <none> to ebfe220f6eb74261832c72711de5103a (127.10.250.65). New cstate: current_term: 1 leader_uuid: "ebfe220f6eb74261832c72711de5103a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793277e296e4f6792df9fc1c46b0d00" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75a67649472f4ae599412823668debb5" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 46373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebfe220f6eb74261832c72711de5103a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 38469 } health_report { overall_health: HEALTHY } } }
I20260620 18:40:45.960342 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.936798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:37101 (local address 127.10.250.67:46373)
0620 18:40:45.936838 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:45.936840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:45.941201 (+ 4361us) server_negotiation.cc:407] Connection header received
0620 18:40:45.941304 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:45.941306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:45.941316 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:45.941343 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:45.941541 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.942260 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.959285 (+ 17025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.959888 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.959944 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.959966 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:45.960222 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 18:40:45.960272 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:40:45.960762 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.936630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:46373 (local address 127.10.250.65:37101)
0620 18:40:45.941098 (+ 4468us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:45.941113 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:45.941234 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:45.941403 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:45.941404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:45.941405 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:45.941477 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:45.941478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.942777 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.942778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:45.944396 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:45.944400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.960038 (+ 15638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.960039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:45.960140 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.960143 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:40:45.960607 (+ 464us) client_negotiation.cc:240] Negotiation successful
0620 18:40:45.960662 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4411,"thread_start_us":76,"threads_started":1}
I20260620 18:40:45.961431 22886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0600142f922541cface782b00b97c0c0" candidate_uuid: "ebfe220f6eb74261832c72711de5103a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75a67649472f4ae599412823668debb5" is_pre_election: true
I20260620 18:40:45.961606 22886 raft_consensus.cc:2454] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebfe220f6eb74261832c72711de5103a in term 0.
I20260620 18:40:45.961709 22885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0600142f922541cface782b00b97c0c0" candidate_uuid: "ebfe220f6eb74261832c72711de5103a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75a67649472f4ae599412823668debb5"
I20260620 18:40:45.961752 22885 raft_consensus.cc:3037] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:45.964588 22885 raft_consensus.cc:2454] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebfe220f6eb74261832c72711de5103a in term 1.
I20260620 18:40:45.974623 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:45.971036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.10.250.65:38469)
0620 18:40:45.971081 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:45.971083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:45.971094 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:40:45.971207 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:45.971208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:45.971230 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:45.971300 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:40:45.971495 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.972248 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.973407 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:45.973717 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:45.973775 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:45.973978 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:40:45.974123 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:40:45.974210 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:45.974456 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 18:40:45.974481 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 18:40:45.987819 22885 raft_consensus.cc:1261] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5 [term 1 FOLLOWER]: Refusing update from remote peer ebfe220f6eb74261832c72711de5103a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:40:45.988376 22961 consensus_queue.cc:1035] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [LEADER]: Connected to new peer: Peer: permanent_uuid: "75a67649472f4ae599412823668debb5" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:40:45.991091 22752 raft_consensus.cc:1261] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00 [term 1 FOLLOWER]: Refusing update from remote peer ebfe220f6eb74261832c72711de5103a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:40:45.991461 22961 consensus_queue.cc:1035] T 0600142f922541cface782b00b97c0c0 P ebfe220f6eb74261832c72711de5103a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793277e296e4f6792df9fc1c46b0d00" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:40:45.996397 22964 mvcc.cc:205] Tried to move back new op lower bound from 7298993545162539008 to 7298993545043054592. Current Snapshot: MvccSnapshot[applied={T|T < 7298993545162539008}]
I20260620 18:40:45.996577 22967 mvcc.cc:205] Tried to move back new op lower bound from 7298993545162539008 to 7298993545043054592. Current Snapshot: MvccSnapshot[applied={T|T < 7298993545162539008}]
Jun 20 18:40:45 dist-test-slave-wp6g krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980845, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
I20260620 18:40:46.004882 22965 mvcc.cc:205] Tried to move back new op lower bound from 7298993545162539008 to 7298993545043054592. Current Snapshot: MvccSnapshot[applied={T|T < 7298993545162539008}]
I20260620 18:40:46.007908 22448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:40:46.008031 22448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:40:46.008721 22448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e2f9b1f4baa4646b75d5c09c8811c9c: Sending DeleteTablet for 3 replicas of tablet 0600142f922541cface782b00b97c0c0
I20260620 18:40:46.009141 22866 tablet_service.cc:1514] Processing DeleteTablet for tablet 0600142f922541cface782b00b97c0c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:46 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:41096
I20260620 18:40:46.009213 22732 tablet_service.cc:1514] Processing DeleteTablet for tablet 0600142f922541cface782b00b97c0c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:46 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:53296
I20260620 18:40:46.009438 22594 tablet_service.cc:1514] Processing DeleteTablet for tablet 0600142f922541cface782b00b97c0c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:46 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:56140
I20260620 18:40:46.009989 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 22529
I20260620 18:40:46.014979 22974 tablet_replica.cc:330] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00: stopping tablet replica
I20260620 18:40:46.015142 22974 raft_consensus.cc:2229] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:40:46.015291 22974 raft_consensus.cc:2258] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:46.015874 22974 ts_tablet_manager.cc:1905] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:46.017743 22974 ts_tablet_manager.cc:1918] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:46.017886 22974 log.cc:1196] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/0600142f922541cface782b00b97c0c0
I20260620 18:40:46.018260 22974 ts_tablet_manager.cc:1939] T 0600142f922541cface782b00b97c0c0 P 9793277e296e4f6792df9fc1c46b0d00: Deleting consensus metadata
I20260620 18:40:46.018822 22434 catalog_manager.cc:4841] TS 9793277e296e4f6792df9fc1c46b0d00 (127.10.250.66:44133): tablet 0600142f922541cface782b00b97c0c0 (table test-table [id=5b89bc45f6994a679ab765b2849f9a6c]) successfully deleted
I20260620 18:40:46.022702 22973 tablet_replica.cc:330] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5: stopping tablet replica
I20260620 18:40:46.022944 22973 raft_consensus.cc:2229] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:40:46.023164 22973 raft_consensus.cc:2258] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 18:40:46.030712 22432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:38469 (error 108)
I20260620 18:40:46.030772 22973 ts_tablet_manager.cc:1905] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:46.031913 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 22666
I20260620 18:40:46.033812 22973 ts_tablet_manager.cc:1918] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:46.033903 22973 log.cc:1196] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/0600142f922541cface782b00b97c0c0
I20260620 18:40:46.034480 22973 ts_tablet_manager.cc:1939] T 0600142f922541cface782b00b97c0c0 P 75a67649472f4ae599412823668debb5: Deleting consensus metadata
I20260620 18:40:46.034968 22433 catalog_manager.cc:4841] TS 75a67649472f4ae599412823668debb5 (127.10.250.67:46373): tablet 0600142f922541cface782b00b97c0c0 (table test-table [id=5b89bc45f6994a679ab765b2849f9a6c]) successfully deleted
I20260620 18:40:46.039156 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 22803
I20260620 18:40:46.046132 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 22416
I20260620 18:40:46.056072 11241 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 18:40:46.401602 11241 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 18:40:46.401710 11241 mini_ranger.cc:66] Stopping Ranger...
I20260620 18:40:46.746914 11241 mini_ranger.cc:68] Stopped Ranger
2026-06-20 18:40:46.747 UTC [22002] LOG: received smart shutdown request
2026-06-20 18:40:46.750 UTC [22002] LOG: background worker "logical replication launcher" (PID 22011) exited with exit code 1
2026-06-20 18:40:46.750 UTC [22006] LOG: shutting down
2026-06-20 18:40:46.936 UTC [22002] LOG: database system is shut down
2026-06-20T18:40:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47855 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:40:47 dist-test-slave-wp6g krb5kdc[22985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:40:47 dist-test-slave-wp6g krb5kdc[22985](info): set up 2 sockets
Jun 20 18:40:47 dist-test-slave-wp6g krb5kdc[22985](info): commencing operation
krb5kdc: starting...
W20260620 18:40:48.090643 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:40:48 dist-test-slave-wp6g krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:40:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.250.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.250.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.250.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/rangeradmin_127.10.250.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.250.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/rangeradmin_127.10.250.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.250.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.250.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.250.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/rangerlookup_127.10.250.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.250.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/rangerlookup_127.10.250.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.250.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.250.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.250.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/HTTP_127.10.250.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.250.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/HTTP_127.10.250.84@KRBTEST.COM.keytab.
I20260620 18:40:48.261533 11241 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGJBaTq/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/postgres -l logfile start
2026-06-20 18:40:50.816 UTC [23015] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 18:40:50.816 UTC [23015] LOG: listening on IPv4 address "127.10.250.84", port 58153
2026-06-20 18:40:50.819 UTC [23015] LOG: listening on Unix socket "/tmp/.s.PGSQL.58153"
2026-06-20 18:40:50.840 UTC [23018] LOG: database system was shut down at 2026-06-20 18:40:49 UTC
2026-06-20 18:40:50.844 UTC [23015] LOG: database system is ready to accept connections
I20260620 18:40:51.810418 11241 mini_postgres.cc:96] Postgres bound to 58153
2026-06-20 18:40:51.816 UTC [23026] FATAL: database "slave" does not exist
127.10.250.84:58153 - accepting connections
I20260620 18:40:51.817195 11241 mini_ranger.cc:161] Starting Ranger...
I20260620 18:40:51.832218 11241 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 18:40:52.386802 11241 mini_ranger.cc:87] Created ranger Postgres database
I20260620 18:40:52.386897 11241 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-admin
2026-06-20 18:40:52,662 [I] DB FLAVOR :POSTGRES
2026-06-20 18:40:52,663 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:40:52,663 [I] Checking connection..
2026-06-20 18:40:52,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 18:40:52,937 [I] Checking connection passed.
2026-06-20 18:40:52,937 [I] --------- Verifying version history table ---------
2026-06-20 18:40:52,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:40:53,185 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:40:53,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:40:53,425 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:40:53,425 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 18:40:53,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 18:40:53.669 UTC [23090] WARNING: there is no transaction in progress
2026-06-20 18:40:53,678 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 18:40:53,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:40:53,945 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 18:40:53,945 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:40:53,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 18:40:54,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 18:40:54,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wp6g.c.gcp-upstream.internal','N') ;"
2026-06-20 18:40:54,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:40:54,947 [I] Table x_portal_user does not exist in database ranger
2026-06-20 18:40:54,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:40:55,232 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 18:40:55,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:40:55,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:40:55,763 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 18:40:55,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 18:40:56.904 UTC [23198] WARNING: there is no transaction in progress
2026-06-20 18:40:56.922 UTC [23198] WARNING: there is no transaction in progress
2026-06-20 18:40:56.941 UTC [23198] WARNING: there is no transaction in progress
2026-06-20 18:40:56.958 UTC [23198] WARNING: there is no transaction in progress
2026-06-20 18:40:56.977 UTC [23198] WARNING: there is no transaction in progress
2026-06-20 18:40:57.000 UTC [23198] WARNING: there is no transaction in progress
2026-06-20 18:40:57.063 UTC [23198] WARNING: there is no transaction in progress
2026-06-20 18:40:57.072 UTC [23198] WARNING: there is no transaction in progress
2026-06-20 18:40:57.090 UTC [23198] WARNING: there is no transaction in progress
2026-06-20 18:40:57.103 UTC [23198] WARNING: there is no transaction in progress
2026-06-20 18:40:57.115 UTC [23198] WARNING: there is no transaction in progress
2026-06-20 18:40:57.131 UTC [23198] WARNING: there is no transaction in progress
2026-06-20 18:40:57.142 UTC [23198] WARNING: there is no transaction in progress
2026-06-20 18:40:57.150 UTC [23198] WARNING: there is no transaction in progress
2026-06-20 18:40:57.159 UTC [23198] WARNING: there is no transaction in progress
2026-06-20 18:40:57.167 UTC [23198] WARNING: there is no transaction in progress
2026-06-20 18:40:57,980 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 18:40:57,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 18:40:58,220 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 18:40:58,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:40:58,494 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 18:40:58,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:40:58,751 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 18:40:58,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:40:59,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:40:59,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wp6g.c.gcp-upstream.internal';"
2026-06-20 18:40:59,534 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 18:40:59,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 18:40:59,781 [I] DB_PATCHES have already been applied
I20260620 18:40:59.786983 11241 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGJBaTq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 18:40:59.787055 11241 mini_ranger.cc:193] Using host: 127.10.250.84
I20260620 18:40:59.790175 11241 mini_ranger.cc:227] Ranger admin URL: http://127.10.250.84:48421
Jun 20, 2026 6:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 18:41:00.920331 11241 mini_ranger.cc:160] Time spent starting Ranger: real 9.103s user 0.000s sys 0.009s
Jun 20, 2026 6:41:22 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:41:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 6:41:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:41:22 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:41:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:41:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:41:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:41:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:41:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:41:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:41:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:41:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:41:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:41:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:41:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:41:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:41:22 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:41:23 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 18:41:24.205936 11241 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.250.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.250.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.250.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/rangerkms_127.10.250.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.250.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/rangerkms_127.10.250.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.250.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.250.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.250.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/HTTP_127.10.250.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.250.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/HTTP_127.10.250.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 18:41:24 dist-test-slave-wp6g krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980884, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 18:41:24.288820 11241 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 18:41:24.299039 11241 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 18:41:25.330871 11241 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 18:41:25.330945 11241 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-kms
I20260620 18:41:25.330958 11241 mini_ranger_kms.cc:227] Using postgres at 127.10.250.84:58153
2026-06-20 18:41:25,499 [I] DB FLAVOR :POSTGRES
2026-06-20 18:41:25,499 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:41:25,499 [I] Checking connection
2026-06-20 18:41:25,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 18:41:25,740 [I] connection success
2026-06-20 18:41:25,741 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 18:41:25,741 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 18:41:25,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 18:41:25,998 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 18:41:25,998 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:41:25,998 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 18:41:25,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.250.84:58153/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 18:41:26,284 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 18:41:26.874584 11241 mini_ranger_kms.cc:310] Created kms service
I20260620 18:41:27.103178 11241 mini_ranger_kms.cc:326] Created kudu user
I20260620 18:41:27.152204 11241 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 18:41:27 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980884, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
I20260620 18:41:27.617466 11241 mini_ranger_kms.cc:384] Added ranger policy
I20260620 18:41:27.617588 11241 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGJBaTq/build/release/bin/postgresql.jar:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGJBaTq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 18:41:27.617609 11241 mini_ranger_kms.cc:240] Using host: 127.10.250.84
I20260620 18:41:27.621090 11241 mini_ranger_kms.cc:276] Ranger KMS PID: 23379
I20260620 18:41:27.621186 11241 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.250.84:49459
Jun 20, 2026 6:41:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:41:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:41:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:41:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:41:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:41:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 18:41:29.895011 11241 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.606s user 0.001s sys 0.150s
I20260620 18:41:29.895131 11241 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 18:41:29.895191 11241 mini_ranger_kms.cc:401] 127.10.250.84:49459/kms/v1/keys
Jun 20 18:41:30 dist-test-slave-wp6g krb5kdc[22985](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980890, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.250.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 18:41:30 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980890, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.250.84@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
Jun 20 18:41:30 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980890, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.250.84@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
Jun 20 18:41:33 dist-test-slave-wp6g krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:41:33.085202 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38593
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:38593
--ranger_config_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.250.84:49459/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:33.168817 23429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:33.169051 23429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:33.169076 23429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:33.170414 23429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:33.170470 23429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:33.170485 23429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:33.170497 23429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:33.170528 23429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:33.172243 23429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.250.84:49459/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:38593
--ranger_config_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:33.172531 23429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:33.172796 23429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:33.178385 23434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:33.178550 23437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:33.178444 23429 server_base.cc:1034] running on GCE node
W20260620 18:41:33.178369 23435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:33.178928 23429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:33.179263 23429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:33.180482 23429 hybrid_clock.cc:648] HybridClock initialized: now 1781980893180459 us; error 39 us; skew 500 ppm
Jun 20 18:41:33 dist-test-slave-wp6g krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980893, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:33.183418 23429 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:41:33.184010 23429 webserver.cc:460] Webserver started at http://127.10.250.126:38653/ using document root <none> and password file <none>
I20260620 18:41:33.184284 23429 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:33.184331 23429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:33.184502 23429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:41:33 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980893, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
I20260620 18:41:33.232218 23429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "e72add3952c045c0b212a8a75a3a7d2d"
format_stamp: "Formatted at 2026-06-20 18:41:33 on dist-test-slave-wp6g"
server_key: "565429ab7df796dc3e0d8f0e3ccb634a"
server_key_iv: "e0ccec436aee47cfb2e479015f9762fc"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:33.232679 23429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "e72add3952c045c0b212a8a75a3a7d2d"
format_stamp: "Formatted at 2026-06-20 18:41:33 on dist-test-slave-wp6g"
server_key: "565429ab7df796dc3e0d8f0e3ccb634a"
server_key_iv: "e0ccec436aee47cfb2e479015f9762fc"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:33.234133 23429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 18:41:33.247928 23444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:33.248147 23429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:41:33.248260 23429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "e72add3952c045c0b212a8a75a3a7d2d"
format_stamp: "Formatted at 2026-06-20 18:41:33 on dist-test-slave-wp6g"
server_key: "565429ab7df796dc3e0d8f0e3ccb634a"
server_key_iv: "e0ccec436aee47cfb2e479015f9762fc"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:33.248378 23429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:33.256887 23429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:33.259151 23429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:33.259341 23429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:33.264452 23429 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:38593
I20260620 18:41:33.264483 23496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:38593 every 8 connection(s)
I20260620 18:41:33.264890 23429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:33.265596 23497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:33.268015 23497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d: Bootstrap starting.
I20260620 18:41:33.268622 23497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:33.268929 23497 log.cc:824] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:33.269620 23497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d: No bootstrap required, opened a new log
I20260620 18:41:33.271472 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 23429
I20260620 18:41:33.271585 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:41:33.271790 11241 external_mini_cluster.cc:1428] Setting key 7c7e038157ddbcf61427a52416e14960
I20260620 18:41:33.272039 23497 raft_consensus.cc:348] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72add3952c045c0b212a8a75a3a7d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38593 } }
I20260620 18:41:33.272286 23497 raft_consensus.cc:374] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:33.272413 23497 raft_consensus.cc:729] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72add3952c045c0b212a8a75a3a7d2d, State: Initialized, Role: FOLLOWER
I20260620 18:41:33.272640 23497 consensus_queue.cc:260] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72add3952c045c0b212a8a75a3a7d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38593 } }
I20260620 18:41:33.272753 23497 raft_consensus.cc:388] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:33.272816 23497 raft_consensus.cc:482] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:33.272876 23497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:33.273679 23497 raft_consensus.cc:504] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72add3952c045c0b212a8a75a3a7d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38593 } }
I20260620 18:41:33.273810 23497 leader_election.cc:302] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e72add3952c045c0b212a8a75a3a7d2d; no voters:
I20260620 18:41:33.273968 23497 leader_election.cc:288] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:33.274129 23503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:33.274241 23497 sys_catalog.cc:564] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:33.274348 23503 raft_consensus.cc:686] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 1 LEADER]: Becoming Leader. State: Replica: e72add3952c045c0b212a8a75a3a7d2d, State: Running, Role: LEADER
I20260620 18:41:33.274487 23503 consensus_queue.cc:237] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72add3952c045c0b212a8a75a3a7d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38593 } }
I20260620 18:41:33.274894 23503 sys_catalog.cc:455] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e72add3952c045c0b212a8a75a3a7d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72add3952c045c0b212a8a75a3a7d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38593 } } }
I20260620 18:41:33.275046 23503 sys_catalog.cc:458] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:33.275209 23503 sys_catalog.cc:455] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e72add3952c045c0b212a8a75a3a7d2d. Latest consensus state: current_term: 1 leader_uuid: "e72add3952c045c0b212a8a75a3a7d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72add3952c045c0b212a8a75a3a7d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38593 } } }
I20260620 18:41:33.275722 23497 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:41:33.275774 23511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:33.276575 23503 sys_catalog.cc:458] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:33.276880 23511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 18:41:33 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:41:33.283641 23511 catalog_manager.cc:1347] Generated new cluster ID: 9fcc7136f6b34c739479c4aaa5ec24dd
I20260620 18:41:33.283779 23511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:33.286698 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.272362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.10.250.126:38593)
0620 18:41:33.272579 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:33.272591 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:33.272642 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:41:33.272800 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:33.272809 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:33.272868 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:33.273012 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:33.273788 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.275292 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.276298 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.276972 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.277092 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.280730 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:33.280781 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:33.280782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:33.280904 (+ 122us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:33.284355 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:33.284840 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:33.284843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:33.284844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:33.284974 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:33.285192 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:33.285193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:33.285193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:33.285583 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:33.286116 (+ 533us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:33.286210 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 18:41:33.286399 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":96,"threads_started":1}
I20260620 18:41:33.298789 23511 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:33.299616 23511 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:33.305171 23511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d: Generated new TSK 0
I20260620 18:41:33.305495 23511 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:41:34.013708 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.250.84:49459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38593
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:34.105540 23542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:34.105924 23542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:34.106009 23542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:34.107513 23542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:34.107587 23542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:34.107636 23542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:41:34.109247 23542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.250.84:49459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:34.109550 23542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:34.109822 23542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:34.116696 23550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:34.116667 23548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:34.116668 23547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:34.117075 23542 server_base.cc:1034] running on GCE node
I20260620 18:41:34.117295 23542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:34.117579 23542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:34.118741 23542 hybrid_clock.cc:648] HybridClock initialized: now 1781980894118725 us; error 38 us; skew 500 ppm
Jun 20 18:41:34 dist-test-slave-wp6g krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980894, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:34.121771 23542 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:41:34.122301 23542 webserver.cc:460] Webserver started at http://127.10.250.65:46849/ using document root <none> and password file <none>
I20260620 18:41:34.122630 23542 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:34.122704 23542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:34.122864 23542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:41:34 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980894, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
I20260620 18:41:34.137640 23542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "ba82114bbb18450c87506b2d76dc592d"
format_stamp: "Formatted at 2026-06-20 18:41:34 on dist-test-slave-wp6g"
server_key: "10ec4eebd71b1252c1cdf9c9e7a58d57"
server_key_iv: "447a2458339397fe6e6ae49e0c3bf7a0"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:34.138264 23542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "ba82114bbb18450c87506b2d76dc592d"
format_stamp: "Formatted at 2026-06-20 18:41:34 on dist-test-slave-wp6g"
server_key: "10ec4eebd71b1252c1cdf9c9e7a58d57"
server_key_iv: "447a2458339397fe6e6ae49e0c3bf7a0"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:34.140453 23542 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 18:41:34.154654 23557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:34.154911 23542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:34.154996 23542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "ba82114bbb18450c87506b2d76dc592d"
format_stamp: "Formatted at 2026-06-20 18:41:34 on dist-test-slave-wp6g"
server_key: "10ec4eebd71b1252c1cdf9c9e7a58d57"
server_key_iv: "447a2458339397fe6e6ae49e0c3bf7a0"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:34.155090 23542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:34.171844 23542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:34.174679 23542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:34.174930 23542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:34.175272 23542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:34.175666 23542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:34.175731 23542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:34.175781 23542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:34.175813 23542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:34.182127 23542 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:35595
I20260620 18:41:34.182682 23542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:41:34.188369 23670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:35595 every 8 connection(s)
I20260620 18:41:34.190081 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 23542
I20260620 18:41:34.190213 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:41:34.190407 11241 external_mini_cluster.cc:1428] Setting key 3ac664c1fd313878ebe7d3e3cd8fa77d
Jun 20 18:41:34 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980894, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
I20260620 18:41:34.204084 23674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:34.190661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:48749 (local address 127.10.250.126:38593)
0620 18:41:34.190848 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:34.190852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:34.192287 (+ 1435us) server_negotiation.cc:407] Connection header received
0620 18:41:34.192322 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:34.192323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:34.192340 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:34.192405 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:34.193376 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:34.194292 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:34.196280 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:34.196669 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:34.196745 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:34.199952 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:34.199954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:34.199954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:34.199962 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:34.202165 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:34.202791 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:34.202800 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:34.202801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:34.202876 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:34.203240 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:34.203241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:34.203241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:34.203381 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:34.203464 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:34.203813 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 18:41:34.203976 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":121,"threads_started":1}
I20260620 18:41:34.204245 23673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:34.189211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38593 (local address 127.10.250.65:48749)
0620 18:41:34.191064 (+ 1853us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:34.191258 (+ 194us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:34.192125 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:34.192549 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:34.192554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:34.192874 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:34.193239 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:34.193248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:34.194416 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:34.194418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:34.196156 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:34.196160 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:34.196839 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:34.196840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:34.197013 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:34.197701 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:34.197715 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:34.199797 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:34.202286 (+ 2489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:34.202309 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:34.202312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:34.202693 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:34.202962 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:34.202963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:34.202963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:34.203137 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:34.203468 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:34.203475 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:34.203642 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 18:41:34.203771 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 18:41:34.203964 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1798,"thread_start_us":143,"threads_started":1}
I20260620 18:41:34.204928 23671 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38593
I20260620 18:41:34.205076 23671 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:34.205295 23671 heartbeater.cc:511] Master 127.10.250.126:38593 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
I20260620 18:41:34.205969 23461 ts_manager.cc:194] Registered new tserver with Master: ba82114bbb18450c87506b2d76dc592d (127.10.250.65:35595)
I20260620 18:41:34.207145 23461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:48749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:41:34.222395 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.250.84:49459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38593
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:34.359833 23677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:34.360236 23677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:34.360397 23677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:34.362411 23677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:34.362593 23677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:34.362705 23677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:41:34.365203 23677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.250.84:49459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:34.365819 23677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:34.366168 23677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:34.373044 23686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:34.373046 23687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:34.373996 23689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:34.374660 23677 server_base.cc:1034] running on GCE node
I20260620 18:41:34.374998 23677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:34.375334 23677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:34.378566 23677 hybrid_clock.cc:648] HybridClock initialized: now 1781980894378536 us; error 41 us; skew 500 ppm
Jun 20 18:41:34 dist-test-slave-wp6g krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980894, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:34.382086 23677 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:41:34.382758 23677 webserver.cc:460] Webserver started at http://127.10.250.66:36841/ using document root <none> and password file <none>
I20260620 18:41:34.383056 23677 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:34.383123 23677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:34.383273 23677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:41:34 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980894, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
I20260620 18:41:34.401594 23677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "ea188831866c4b768fcbdb44af1bf97e"
format_stamp: "Formatted at 2026-06-20 18:41:34 on dist-test-slave-wp6g"
server_key: "b0fee74f9e0a1216bf37ffccf3a6a674"
server_key_iv: "58c5566217512fb06e9d413c13d67b62"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:34.402210 23677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "ea188831866c4b768fcbdb44af1bf97e"
format_stamp: "Formatted at 2026-06-20 18:41:34 on dist-test-slave-wp6g"
server_key: "b0fee74f9e0a1216bf37ffccf3a6a674"
server_key_iv: "58c5566217512fb06e9d413c13d67b62"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:34.407931 23677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.001s
I20260620 18:41:34.424986 23696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:34.425248 23677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:41:34.425340 23677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "ea188831866c4b768fcbdb44af1bf97e"
format_stamp: "Formatted at 2026-06-20 18:41:34 on dist-test-slave-wp6g"
server_key: "b0fee74f9e0a1216bf37ffccf3a6a674"
server_key_iv: "58c5566217512fb06e9d413c13d67b62"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:34.425436 23677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:34.459098 23677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:34.461848 23677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:34.462100 23677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:34.462424 23677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:34.462864 23677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:34.462960 23677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:34.463032 23677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:34.463088 23677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:34.469475 23677 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:44497
I20260620 18:41:34.470011 23677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:41:34 dist-test-slave-wp6g krb5kdc[22985](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980894, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:34.472915 23809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:44497 every 8 connection(s)
I20260620 18:41:34.476884 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 23677
I20260620 18:41:34.476994 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:41:34.477185 11241 external_mini_cluster.cc:1428] Setting key 9ad4cd65b420383c951dd5e6d98c8c5e
Jun 20 18:41:34 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980894, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:41:34.486169 23674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:34.473802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:38861 (local address 127.10.250.126:38593)
0620 18:41:34.473859 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:34.473861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:34.474168 (+ 307us) server_negotiation.cc:407] Connection header received
0620 18:41:34.475569 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:34.475570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:34.475596 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:34.475627 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:34.476512 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:34.477312 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:34.479289 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:34.479611 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:34.479658 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:34.482294 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:34.482296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:34.482296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:34.482305 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:34.484145 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:34.484976 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:34.484977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:34.484978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:34.485043 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:34.485381 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:34.485382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:34.485382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:34.485526 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:34.485566 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:34.485966 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 18:41:34.486067 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:41:34.487022 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:34.473502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38593 (local address 127.10.250.66:38861)
0620 18:41:34.474037 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:34.474083 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:34.475440 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:34.475872 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:34.475877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:34.476115 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:34.476397 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:34.476404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:34.477487 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:34.477488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:34.479176 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:34.479180 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:34.479774 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:34.479775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:34.479919 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:34.480533 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:34.480545 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:34.482178 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:34.484258 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:34.484266 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:34.484269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:34.484876 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:34.485153 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:34.485155 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:34.485155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:34.485306 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:34.485618 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:34.485625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:34.485813 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 18:41:34.486571 (+ 758us) client_negotiation.cc:240] Negotiation successful
0620 18:41:34.486772 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":86,"threads_started":1}
I20260620 18:41:34.487715 23810 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38593
I20260620 18:41:34.487831 23810 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:34.488026 23810 heartbeater.cc:511] Master 127.10.250.126:38593 requested a full tablet report, sending...
I20260620 18:41:34.488451 23461 ts_manager.cc:194] Registered new tserver with Master: ea188831866c4b768fcbdb44af1bf97e (127.10.250.66:44497)
I20260620 18:41:34.488951 23461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:38861
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:41:34.533869 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.250.84:49459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38593
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:34.649235 23815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:34.649605 23815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:34.649735 23815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:34.651755 23815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:34.651883 23815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:34.651997 23815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:41:34.654600 23815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.250.84:49459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:34.655130 23815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:34.655489 23815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:34.662915 23822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:34.663121 23815 server_base.cc:1034] running on GCE node
W20260620 18:41:34.663254 23824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:34.663606 23821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:34.663969 23815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:34.664266 23815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:34.665462 23815 hybrid_clock.cc:648] HybridClock initialized: now 1781980894665439 us; error 38 us; skew 500 ppm
Jun 20 18:41:34 dist-test-slave-wp6g krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980894, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:34.668898 23815 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:41:34.669456 23815 webserver.cc:460] Webserver started at http://127.10.250.67:38375/ using document root <none> and password file <none>
I20260620 18:41:34.669721 23815 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:34.669791 23815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:34.669930 23815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:41:34 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980894, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
I20260620 18:41:34.692773 23815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "7ac2bc8f2d1b41eba6803cef2b20b659"
format_stamp: "Formatted at 2026-06-20 18:41:34 on dist-test-slave-wp6g"
server_key: "d54ee3bc234298975953d096c7fbb7de"
server_key_iv: "fa64e0a1719cde4f2df7576290c37010"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:34.693274 23815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "7ac2bc8f2d1b41eba6803cef2b20b659"
format_stamp: "Formatted at 2026-06-20 18:41:34 on dist-test-slave-wp6g"
server_key: "d54ee3bc234298975953d096c7fbb7de"
server_key_iv: "fa64e0a1719cde4f2df7576290c37010"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:34.702106 23815 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.004s
I20260620 18:41:34.730654 23831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:34.731077 23815 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.000s
I20260620 18:41:34.731210 23815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "7ac2bc8f2d1b41eba6803cef2b20b659"
format_stamp: "Formatted at 2026-06-20 18:41:34 on dist-test-slave-wp6g"
server_key: "d54ee3bc234298975953d096c7fbb7de"
server_key_iv: "fa64e0a1719cde4f2df7576290c37010"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:34.731357 23815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:34.759727 23815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:34.762432 23815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:34.762682 23815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:34.763012 23815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:34.763442 23815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:34.763522 23815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:34.763588 23815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:34.763643 23815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:34.769945 23815 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:39937
I20260620 18:41:34.770462 23815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:41:34.772553 23944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:39937 every 8 connection(s)
I20260620 18:41:34.773522 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 23815
I20260620 18:41:34.773653 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:41:34.773864 11241 external_mini_cluster.cc:1428] Setting key ff64c9960968b2bd7379fabcedd19df4
Jun 20 18:41:34 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980894, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:41:34.786188 23674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:34.774330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:54285 (local address 127.10.250.126:38593)
0620 18:41:34.774423 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:34.774425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:34.774435 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:41:34.775509 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:34.775510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:34.775521 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:34.775550 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:34.776566 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:34.777346 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:34.778866 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:34.779160 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:34.779203 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:34.781959 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:34.781960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:34.781960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:34.781969 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:34.783801 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:34.784412 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:34.784413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:34.784414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:34.784478 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:34.784781 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:34.784782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:34.784783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:34.784921 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:34.784962 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:34.786002 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0620 18:41:34.786095 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:41:34.787118 23947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:34.773925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38593 (local address 127.10.250.67:54285)
0620 18:41:34.774281 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:34.774366 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:34.775363 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:34.775750 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:34.775754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:34.776148 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:34.776453 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:34.776460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:34.777464 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:34.777465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:34.778763 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:34.778766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:34.779326 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:34.779327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:34.779474 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:34.780116 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:34.780127 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:34.781832 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:34.783927 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:34.783936 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:34.783939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:34.784315 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:34.784554 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:34.784555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:34.784555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:34.784707 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:34.785008 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:34.785014 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:34.785827 (+ 813us) client_negotiation.cc:769] Sending connection context
0620 18:41:34.786683 (+ 856us) client_negotiation.cc:240] Negotiation successful
0620 18:41:34.786871 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":95,"threads_started":1}
I20260620 18:41:34.787755 23945 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38593
I20260620 18:41:34.787889 23945 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:34.788162 23945 heartbeater.cc:511] Master 127.10.250.126:38593 requested a full tablet report, sending...
I20260620 18:41:34.788555 23461 ts_manager.cc:194] Registered new tserver with Master: 7ac2bc8f2d1b41eba6803cef2b20b659 (127.10.250.67:39937)
I20260620 18:41:34.789031 23461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:54285
I20260620 18:41:34.796420 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:41:34.797086 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 23542
I20260620 18:41:34.803009 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 23677
I20260620 18:41:34.810585 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 23815
I20260620 18:41:34.817538 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 23429
I20260620 18:41:34.823750 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38593
--webserver_interface=127.10.250.126
--webserver_port=38653
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:38593
--ranger_config_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.250.84:49459/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.250.84:49459/kms with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:34.911801 23948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:34.912082 23948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:34.912127 23948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:34.913450 23948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:34.913543 23948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:34.913568 23948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:34.913590 23948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:34.913618 23948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:34.915561 23948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.250.84:49459/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:38593
--ranger_config_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=38653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:34.916107 23948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:34.916409 23948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:34.922842 23954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:34.922886 23956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:34.922885 23953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:34.923188 23948 server_base.cc:1034] running on GCE node
I20260620 18:41:34.923517 23948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:34.923974 23948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:34.925217 23948 hybrid_clock.cc:648] HybridClock initialized: now 1781980894925188 us; error 43 us; skew 500 ppm
Jun 20 18:41:34 dist-test-slave-wp6g krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980894, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:34.928277 23948 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:41:34.928792 23948 webserver.cc:460] Webserver started at http://127.10.250.126:38653/ using document root <none> and password file <none>
I20260620 18:41:34.929049 23948 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:34.929131 23948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:41:34 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980894, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
I20260620 18:41:35.005869 23948 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 18:41:35.006738 23964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:35.006997 23948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:41:35.007126 23948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "e72add3952c045c0b212a8a75a3a7d2d"
format_stamp: "Formatted at 2026-06-20 18:41:33 on dist-test-slave-wp6g"
server_key: "565429ab7df796dc3e0d8f0e3ccb634a"
server_key_iv: "e0ccec436aee47cfb2e479015f9762fc"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:35.007648 23948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:35.032665 23948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:35.035467 23948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:35.035640 23948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:35.041092 23948 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:38593
I20260620 18:41:35.041188 24016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:38593 every 8 connection(s)
I20260620 18:41:35.041611 23948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:35.041915 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 23948
I20260620 18:41:35.042227 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.250.84:49459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:35595
--local_ip_for_outbound_sockets=127.10.250.65
--tserver_master_addrs=127.10.250.126:38593
--webserver_port=46849
--webserver_interface=127.10.250.65
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.250.84:49459/kms with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:35.043751 24017 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:41:35.044462 24017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d: Bootstrap starting.
I20260620 18:41:35.053218 24017 log.cc:824] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:35.055296 24017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:41:35.055652 24017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d: Bootstrap complete.
I20260620 18:41:35.058172 24017 raft_consensus.cc:348] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72add3952c045c0b212a8a75a3a7d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38593 } }
I20260620 18:41:35.058458 24017 raft_consensus.cc:729] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72add3952c045c0b212a8a75a3a7d2d, State: Initialized, Role: FOLLOWER
I20260620 18:41:35.058707 24017 consensus_queue.cc:260] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72add3952c045c0b212a8a75a3a7d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38593 } }
I20260620 18:41:35.058805 24017 raft_consensus.cc:388] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:35.058856 24017 raft_consensus.cc:482] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:35.058909 24017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:41:35.059994 24017 raft_consensus.cc:504] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72add3952c045c0b212a8a75a3a7d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38593 } }
I20260620 18:41:35.060124 24017 leader_election.cc:302] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e72add3952c045c0b212a8a75a3a7d2d; no voters:
I20260620 18:41:35.060339 24017 leader_election.cc:288] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:41:35.060465 24021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:41:35.060614 24021 raft_consensus.cc:686] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [term 2 LEADER]: Becoming Leader. State: Replica: e72add3952c045c0b212a8a75a3a7d2d, State: Running, Role: LEADER
I20260620 18:41:35.060642 24017 sys_catalog.cc:564] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:35.060824 24021 consensus_queue.cc:237] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72add3952c045c0b212a8a75a3a7d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38593 } }
I20260620 18:41:35.061167 24021 sys_catalog.cc:455] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e72add3952c045c0b212a8a75a3a7d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72add3952c045c0b212a8a75a3a7d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38593 } } }
I20260620 18:41:35.061332 24021 sys_catalog.cc:458] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:35.061529 24021 sys_catalog.cc:455] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e72add3952c045c0b212a8a75a3a7d2d. Latest consensus state: current_term: 2 leader_uuid: "e72add3952c045c0b212a8a75a3a7d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72add3952c045c0b212a8a75a3a7d2d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38593 } } }
I20260620 18:41:35.061638 24021 sys_catalog.cc:458] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:35.061817 24030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:35.062263 24017 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:41:35.062458 24030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:35.069129 24030 catalog_manager.cc:1259] Loaded cluster ID: 9fcc7136f6b34c739479c4aaa5ec24dd
I20260620 18:41:35.069329 24030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:35.071251 24030 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:35.071661 24030 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d: Loaded TSK: 0
I20260620 18:41:35.072048 24030 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:41:35.165361 24019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:35.165681 24019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:35.165822 24019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:35.167764 24019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:35.168021 24019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:35.168116 24019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:41:35.170506 24019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.250.84:49459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:35595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=46849
--tserver_master_addrs=127.10.250.126:38593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:35.171105 24019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:35.171453 24019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:35.178236 24054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:35.178814 24057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:35.178956 24055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:35.180027 24019 server_base.cc:1034] running on GCE node
I20260620 18:41:35.180263 24019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:35.180533 24019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:35.181712 24019 hybrid_clock.cc:648] HybridClock initialized: now 1781980895181687 us; error 37 us; skew 500 ppm
Jun 20 18:41:35 dist-test-slave-wp6g krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980895, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:35.185321 24019 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:41:35.186178 24019 webserver.cc:460] Webserver started at http://127.10.250.65:46849/ using document root <none> and password file <none>
I20260620 18:41:35.186483 24019 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:35.186760 24019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:41:35 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980895, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
I20260620 18:41:35.218786 24019 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:41:35.220253 24065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:35.220980 24019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 18:41:35.221089 24019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "ba82114bbb18450c87506b2d76dc592d"
format_stamp: "Formatted at 2026-06-20 18:41:34 on dist-test-slave-wp6g"
server_key: "10ec4eebd71b1252c1cdf9c9e7a58d57"
server_key_iv: "447a2458339397fe6e6ae49e0c3bf7a0"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:35.221729 24019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:35.241636 24019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:35.245020 24019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:35.245291 24019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:35.245592 24019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:35.246150 24019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:35.246261 24019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:35.246369 24019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:35.246441 24019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:35.253605 24019 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:35595
I20260620 18:41:35.254403 24019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:41:35.259331 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 24019
I20260620 18:41:35.259665 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.250.84:49459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:44497
--local_ip_for_outbound_sockets=127.10.250.66
--tserver_master_addrs=127.10.250.126:38593
--webserver_port=36841
--webserver_interface=127.10.250.66
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.250.84:49459/kms with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:41:35 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980895, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:41:35.266970 24178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:35595 every 8 connection(s)
I20260620 18:41:35.270969 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:35.256172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:47121 (local address 127.10.250.126:38593)
0620 18:41:35.257864 (+ 1692us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:35.257874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:35.257911 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:41:35.258068 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:35.258074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:35.258130 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:35.258278 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:35.259186 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:35.261098 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:35.262646 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:35.263270 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:35.263395 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:35.266271 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:35.266277 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:35.266278 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:35.266291 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:35.268500 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:35.269032 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:35.269035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:35.269036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:35.269155 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:35.269456 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:35.269456 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:35.269457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:35.269766 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:35.269847 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:35.270618 (+ 771us) server_negotiation.cc:299] Negotiation successful
0620 18:41:35.270742 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1633,"thread_start_us":104,"threads_started":1}
I20260620 18:41:35.271517 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:35.255777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38593 (local address 127.10.250.65:47121)
0620 18:41:35.256418 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:35.256472 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:35.257632 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:35.258458 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:35.258462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:35.258738 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:35.259045 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:35.259052 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:35.261243 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:35.261245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:35.262478 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:35.262481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:35.263569 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:35.263571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:35.263719 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:35.264412 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:35.264423 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:35.266093 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:35.268610 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:35.268617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:35.268620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:35.268930 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:35.269252 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:35.269252 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:35.269253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:35.269383 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:35.269877 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:35.269883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:35.270036 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 18:41:35.270140 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:41:35.270311 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":87,"threads_started":1}
I20260620 18:41:35.272521 24179 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38593
I20260620 18:41:35.272614 24179 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:35.272802 24179 heartbeater.cc:511] Master 127.10.250.126:38593 requested a full tablet report, sending...
W20260620 18:41:35.273677 24179 heartbeater.cc:644] Failed to heartbeat to 127.10.250.126:38593 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:41:35.273746 24179 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:35.273806 24179 heartbeater.cc:511] Master 127.10.250.126:38593 requested a full tablet report, sending...
I20260620 18:41:35.274192 24179 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:35.274274 24179 heartbeater.cc:511] Master 127.10.250.126:38593 requested a full tablet report, sending...
W20260620 18:41:35.274643 24179 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:41:35.396358 24183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:35.396777 24183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:35.396893 24183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:35.399019 24183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:35.399147 24183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:35.399263 24183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:41:35.401691 24183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.250.84:49459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:44497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=36841
--tserver_master_addrs=127.10.250.126:38593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:35.402253 24183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:35.402616 24183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:35.410531 24191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:35.410997 24188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:35.411330 24189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:35.412397 24183 server_base.cc:1034] running on GCE node
I20260620 18:41:35.412676 24183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:35.413021 24183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:35.414215 24183 hybrid_clock.cc:648] HybridClock initialized: now 1781980895414186 us; error 41 us; skew 500 ppm
Jun 20 18:41:35 dist-test-slave-wp6g krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980895, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:35.417894 24183 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:41:35.418650 24183 webserver.cc:460] Webserver started at http://127.10.250.66:36841/ using document root <none> and password file <none>
I20260620 18:41:35.419023 24183 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:35.419163 24183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:41:35 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980895, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
I20260620 18:41:35.441051 24183 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:41:35.442394 24199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:35.442852 24183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:35.442951 24183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "ea188831866c4b768fcbdb44af1bf97e"
format_stamp: "Formatted at 2026-06-20 18:41:34 on dist-test-slave-wp6g"
server_key: "b0fee74f9e0a1216bf37ffccf3a6a674"
server_key_iv: "58c5566217512fb06e9d413c13d67b62"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:35.443449 24183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:35.486631 24183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:35.489336 24183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:35.489562 24183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:35.489823 24183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:35.490242 24183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:35.490322 24183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:35.490392 24183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:35.490458 24183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:35.497833 24183 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:44497
I20260620 18:41:35.498433 24183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:41:35.499030 24312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:44497 every 8 connection(s)
I20260620 18:41:35.500181 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 24183
I20260620 18:41:35.500538 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.250.84:49459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:39937
--local_ip_for_outbound_sockets=127.10.250.67
--tserver_master_addrs=127.10.250.126:38593
--webserver_port=38375
--webserver_interface=127.10.250.67
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.250.84:49459/kms with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:41:35 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980895, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:41:35.513340 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:35.500255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:56525 (local address 127.10.250.126:38593)
0620 18:41:35.500306 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:35.500308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:35.500952 (+ 644us) server_negotiation.cc:407] Connection header received
0620 18:41:35.501695 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:35.501696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:35.501714 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:35.501747 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:35.502686 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:35.503775 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:35.505447 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:35.505777 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:35.505821 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:35.508582 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:35.508584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:35.508584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:35.508593 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:35.510355 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:35.510982 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:35.510984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:35.510984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:35.511049 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:35.511386 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:35.511387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:35.511387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:35.511532 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:35.511593 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:35.513114 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0620 18:41:35.513234 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 18:41:35.514248 24315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:35.499873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38593 (local address 127.10.250.66:56525)
0620 18:41:35.500812 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:35.500860 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:35.501515 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:35.501945 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:35.501949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:35.502250 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:35.502573 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:35.502581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:35.503924 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:35.503925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:35.505285 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:35.505288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:35.505937 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:35.505938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:35.506087 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:35.506792 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:35.506804 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:35.508450 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:35.510457 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:35.510466 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:35.510469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:35.510883 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:35.511134 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:35.511135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:35.511135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:35.511282 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:35.511935 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:35.511942 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:35.512118 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 18:41:35.513787 (+ 1669us) client_negotiation.cc:240] Negotiation successful
0620 18:41:35.513993 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":91,"threads_started":1}
I20260620 18:41:35.516203 24313 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38593
I20260620 18:41:35.516323 24313 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:35.516558 24313 heartbeater.cc:511] Master 127.10.250.126:38593 requested a full tablet report, sending...
W20260620 18:41:35.517416 24313 heartbeater.cc:644] Failed to heartbeat to 127.10.250.126:38593 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:41:35.517503 24313 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:35.517575 24313 heartbeater.cc:511] Master 127.10.250.126:38593 requested a full tablet report, sending...
I20260620 18:41:35.518416 24313 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:35.518544 24313 heartbeater.cc:511] Master 127.10.250.126:38593 requested a full tablet report, sending...
W20260620 18:41:35.519001 24313 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:41:35.640112 24316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:35.640506 24316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:35.640691 24316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:35.642853 24316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:35.642985 24316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:35.643070 24316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:41:35.645700 24316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.250.84:49459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:39937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=38375
--tserver_master_addrs=127.10.250.126:38593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:35.646262 24316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:35.646639 24316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:35.655514 24321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:35.655802 24322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:35.656469 24316 server_base.cc:1034] running on GCE node
W20260620 18:41:35.658653 24324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:35.658977 24316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:35.659302 24316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:35.660485 24316 hybrid_clock.cc:648] HybridClock initialized: now 1781980895660452 us; error 50 us; skew 500 ppm
Jun 20 18:41:35 dist-test-slave-wp6g krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980895, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:35.664119 24316 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:41:35.664711 24316 webserver.cc:460] Webserver started at http://127.10.250.67:38375/ using document root <none> and password file <none>
I20260620 18:41:35.665025 24316 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:35.665108 24316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:41:35 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980895, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
I20260620 18:41:35.689996 24316 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:35.691112 24333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:35.691502 24316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:41:35.691620 24316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "7ac2bc8f2d1b41eba6803cef2b20b659"
format_stamp: "Formatted at 2026-06-20 18:41:34 on dist-test-slave-wp6g"
server_key: "d54ee3bc234298975953d096c7fbb7de"
server_key_iv: "fa64e0a1719cde4f2df7576290c37010"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:35.692170 24316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:35.707813 24316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:35.710434 24316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:35.710667 24316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:35.710976 24316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:35.711392 24316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:35.711441 24316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:35.711476 24316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:35.711494 24316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:35.719455 24316 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:39937
I20260620 18:41:35.720010 24316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:41:35.722929 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 24316
I20260620 18:41:35.723621 24446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:39937 every 8 connection(s)
Jun 20 18:41:35 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980895, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:41:35.733152 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:35.723449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.10.250.126:38593)
0620 18:41:35.723562 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:35.723565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:35.724197 (+ 632us) server_negotiation.cc:407] Connection header received
0620 18:41:35.724319 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:35.724320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:35.724333 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:35.724368 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:35.725240 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:35.725963 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:35.727162 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:35.727459 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:35.727544 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:35.729415 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:35.729417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:35.729417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:35.729424 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:35.731304 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:35.731935 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:35.731937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:35.731937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:35.732007 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:35.732236 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:35.732237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:35.732237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:35.732705 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:35.732910 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:35.732957 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 18:41:35.733046 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:41:35.736981 24451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:35.724753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:58013 (local address 127.10.250.126:38593)
0620 18:41:35.724894 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:35.724896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:35.725843 (+ 947us) server_negotiation.cc:407] Connection header received
0620 18:41:35.725874 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:35.725875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:35.725890 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:35.725963 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:35.726870 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:35.727571 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:35.728911 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:35.729123 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:35.729221 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:35.731988 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:35.731989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:35.731989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:35.731996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:35.735103 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:35.735703 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:35.735704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:35.735704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:35.735768 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:35.736131 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:35.736132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:35.736132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:35.736257 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:35.736368 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:35.736791 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 18:41:35.736895 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260620 18:41:35.737205 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:35.724412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38593 (local address 127.10.250.67:58013)
0620 18:41:35.724934 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:35.724989 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:35.725683 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:35.726110 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:35.726115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:35.726368 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:35.726744 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:35.726752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:35.727672 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:35.727673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:35.728792 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:35.728794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:35.729253 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:35.729254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:35.729434 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:35.730066 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:35.730087 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:35.731850 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:35.735211 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:35.735224 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:35.735227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:35.735598 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:35.735846 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:35.735847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:35.735847 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:35.736027 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:35.736367 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:35.736375 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:35.736590 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 18:41:35.736725 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 18:41:35.736923 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":120,"threads_started":1}
I20260620 18:41:35.738157 24447 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38593
I20260620 18:41:35.738322 24447 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:35.738617 24447 heartbeater.cc:511] Master 127.10.250.126:38593 requested a full tablet report, sending...
W20260620 18:41:35.739411 24447 heartbeater.cc:644] Failed to heartbeat to 127.10.250.126:38593 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:41:35.739507 24447 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:35.739584 24447 heartbeater.cc:511] Master 127.10.250.126:38593 requested a full tablet report, sending...
I20260620 18:41:35.740083 24447 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:35.740176 24447 heartbeater.cc:511] Master 127.10.250.126:38593 requested a full tablet report, sending...
W20260620 18:41:35.740579 24447 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:41:36.274824 24179 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:36.274977 24179 heartbeater.cc:511] Master 127.10.250.126:38593 requested a full tablet report, sending...
I20260620 18:41:36.275470 23981 ts_manager.cc:194] Registered new tserver with Master: ba82114bbb18450c87506b2d76dc592d (127.10.250.65:35595)
I20260620 18:41:36.276473 23981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:47121
Jun 20 18:41:36 dist-test-slave-wp6g krb5kdc[22985](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980896, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:36.519200 24313 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:36.519390 24313 heartbeater.cc:511] Master 127.10.250.126:38593 requested a full tablet report, sending...
I20260620 18:41:36.519837 23981 ts_manager.cc:194] Registered new tserver with Master: ea188831866c4b768fcbdb44af1bf97e (127.10.250.66:44497)
I20260620 18:41:36.520358 23981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:56525
Jun 20 18:41:36 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980896, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
I20260620 18:41:36.740792 24447 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:36.740970 24447 heartbeater.cc:511] Master 127.10.250.126:38593 requested a full tablet report, sending...
I20260620 18:41:36.741334 23981 ts_manager.cc:194] Registered new tserver with Master: 7ac2bc8f2d1b41eba6803cef2b20b659 (127.10.250.67:39937)
I20260620 18:41:36.741856 23981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:58013
I20260620 18:41:36.741950 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:41:36.752355 24468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.743060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45172 (local address 127.10.250.126:38593)
0620 18:41:36.743318 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:36.743320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:36.743491 (+ 171us) server_negotiation.cc:407] Connection header received
0620 18:41:36.743520 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:36.743521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:36.743531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:36.743609 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:36.744179 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.744996 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.746131 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.746473 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.746568 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:36.748303 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:36.748305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:36.748305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:36.748314 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:36.750414 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:36.751133 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:36.751135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:36.751135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:36.751215 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:36.751422 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:36.751422 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:36.751423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:36.751562 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:36.751650 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:36.752156 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 18:41:36.752246 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":62,"threads_started":1}
I20260620 18:41:36.758906 23981 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:41:36.761525 23981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:41:36.778258 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.769438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46042 (local address 127.10.250.65:35595)
0620 18:41:36.769882 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:36.769888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:36.770375 (+ 487us) server_negotiation.cc:407] Connection header received
0620 18:41:36.770477 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:36.770480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:36.770497 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:36.770587 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:36.770873 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.772406 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.776494 (+ 4088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.777694 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.777753 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:36.777828 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:36.778112 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 18:41:36.778157 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":52,"threads_started":1}
I20260620 18:41:36.778790 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.769182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:35595 (local address 127.0.0.1:46042)
0620 18:41:36.770324 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:36.770331 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:36.770399 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:36.770709 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:36.770712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:36.770714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:36.770785 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:36.770787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.772508 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.772509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:36.776384 (+ 3875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:36.776388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.777897 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.777898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:36.777989 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:36.778005 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 18:41:36.778634 (+ 629us) client_negotiation.cc:240] Negotiation successful
0620 18:41:36.778712 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"thread_start_us":67,"threads_started":1}
I20260620 18:41:36.783414 24113 tablet_service.cc:1467] Processing CreateTablet for tablet 1f2c1087d25648b59dfb3aac6a53e43e (DEFAULT_TABLE table=test-table [id=d69630742eef4e73a2cd0ca6f9f75a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:36.783881 24113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f2c1087d25648b59dfb3aac6a53e43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:36.790922 24475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.769016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:39937 (local address 127.0.0.1:34894)
0620 18:41:36.770173 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:36.770181 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:36.770210 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:36.773564 (+ 3354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:36.773565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:36.773568 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:36.774056 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:36.774059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.779051 (+ 4992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.779052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:36.781598 (+ 2546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:36.781601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.790622 (+ 9021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.790623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:36.790708 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:36.790714 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:41:36.790757 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 18:41:36.790816 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"thread_start_us":67,"threads_started":1}
I20260620 18:41:36.791625 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.769111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34894 (local address 127.10.250.67:39937)
0620 18:41:36.773078 (+ 3967us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:36.773084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:36.773095 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:41:36.773239 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:36.773242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:36.773260 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:36.774137 (+ 877us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:36.774223 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.775379 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.781720 (+ 6341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.789409 (+ 7689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.791357 (+ 1948us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:36.791461 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:36.791491 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 18:41:36.791545 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3811,"thread_start_us":49,"threads_started":1}
I20260620 18:41:36.793004 24374 tablet_service.cc:1467] Processing CreateTablet for tablet 1f2c1087d25648b59dfb3aac6a53e43e (DEFAULT_TABLE table=test-table [id=d69630742eef4e73a2cd0ca6f9f75a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:36.793447 24374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f2c1087d25648b59dfb3aac6a53e43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:36.795426 24479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.768769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:44497 (local address 127.0.0.1:37934)
0620 18:41:36.769955 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:36.770012 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:36.770075 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:36.789523 (+ 19448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:36.789524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:36.789528 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:36.789588 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:36.789590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.792993 (+ 3403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.792994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:36.794129 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:36.794133 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.795170 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.795171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:36.795238 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:36.795244 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:41:36.795278 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 18:41:36.795328 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"thread_start_us":88,"threads_started":1}
I20260620 18:41:36.796329 24476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.768859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.10.250.66:44497)
0620 18:41:36.789094 (+ 20235us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:36.789099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:36.789114 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:41:36.789154 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:36.789156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:36.789174 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:36.789663 (+ 489us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:36.789757 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.792887 (+ 3130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.794242 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.795060 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.796093 (+ 1033us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:36.796173 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:36.796204 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 18:41:36.796252 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20193,"thread_start_us":46,"threads_started":1}
I20260620 18:41:36.797071 24247 tablet_service.cc:1467] Processing CreateTablet for tablet 1f2c1087d25648b59dfb3aac6a53e43e (DEFAULT_TABLE table=test-table [id=d69630742eef4e73a2cd0ca6f9f75a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:36.797230 24481 tablet_bootstrap.cc:492] T 1f2c1087d25648b59dfb3aac6a53e43e P ba82114bbb18450c87506b2d76dc592d: Bootstrap starting.
I20260620 18:41:36.797473 24247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f2c1087d25648b59dfb3aac6a53e43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:36.798134 24481 tablet_bootstrap.cc:654] T 1f2c1087d25648b59dfb3aac6a53e43e P ba82114bbb18450c87506b2d76dc592d: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:36.798959 24481 log.cc:824] T 1f2c1087d25648b59dfb3aac6a53e43e P ba82114bbb18450c87506b2d76dc592d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:36.799813 24481 tablet_bootstrap.cc:492] T 1f2c1087d25648b59dfb3aac6a53e43e P ba82114bbb18450c87506b2d76dc592d: No bootstrap required, opened a new log
I20260620 18:41:36.799880 24481 ts_tablet_manager.cc:1397] T 1f2c1087d25648b59dfb3aac6a53e43e P ba82114bbb18450c87506b2d76dc592d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:41:36.802181 24481 raft_consensus.cc:348] T 1f2c1087d25648b59dfb3aac6a53e43e P ba82114bbb18450c87506b2d76dc592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea188831866c4b768fcbdb44af1bf97e" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44497 } } peers { permanent_uuid: "7ac2bc8f2d1b41eba6803cef2b20b659" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 39937 } } peers { permanent_uuid: "ba82114bbb18450c87506b2d76dc592d" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 35595 } }
I20260620 18:41:36.802371 24481 raft_consensus.cc:374] T 1f2c1087d25648b59dfb3aac6a53e43e P ba82114bbb18450c87506b2d76dc592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:36.802402 24481 raft_consensus.cc:729] T 1f2c1087d25648b59dfb3aac6a53e43e P ba82114bbb18450c87506b2d76dc592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba82114bbb18450c87506b2d76dc592d, State: Initialized, Role: FOLLOWER
I20260620 18:41:36.802565 24481 consensus_queue.cc:260] T 1f2c1087d25648b59dfb3aac6a53e43e P ba82114bbb18450c87506b2d76dc592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea188831866c4b768fcbdb44af1bf97e" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44497 } } peers { permanent_uuid: "7ac2bc8f2d1b41eba6803cef2b20b659" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 39937 } } peers { permanent_uuid: "ba82114bbb18450c87506b2d76dc592d" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 35595 } }
I20260620 18:41:36.802899 24179 heartbeater.cc:503] Master 127.10.250.126:38593 was elected leader, sending a full tablet report...
I20260620 18:41:36.802860 24481 ts_tablet_manager.cc:1428] T 1f2c1087d25648b59dfb3aac6a53e43e P ba82114bbb18450c87506b2d76dc592d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:36.803558 24482 tablet_bootstrap.cc:492] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659: Bootstrap starting.
I20260620 18:41:36.804554 24482 tablet_bootstrap.cc:654] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:36.804888 24482 log.cc:824] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:36.805776 24482 tablet_bootstrap.cc:492] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659: No bootstrap required, opened a new log
I20260620 18:41:36.805902 24482 ts_tablet_manager.cc:1397] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:41:36.808468 24482 raft_consensus.cc:348] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea188831866c4b768fcbdb44af1bf97e" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44497 } } peers { permanent_uuid: "7ac2bc8f2d1b41eba6803cef2b20b659" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 39937 } } peers { permanent_uuid: "ba82114bbb18450c87506b2d76dc592d" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 35595 } }
I20260620 18:41:36.808736 24482 raft_consensus.cc:374] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:36.808816 24482 raft_consensus.cc:729] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac2bc8f2d1b41eba6803cef2b20b659, State: Initialized, Role: FOLLOWER
I20260620 18:41:36.809005 24482 consensus_queue.cc:260] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea188831866c4b768fcbdb44af1bf97e" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44497 } } peers { permanent_uuid: "7ac2bc8f2d1b41eba6803cef2b20b659" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 39937 } } peers { permanent_uuid: "ba82114bbb18450c87506b2d76dc592d" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 35595 } }
I20260620 18:41:36.809355 24482 ts_tablet_manager.cc:1428] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:36.809511 24447 heartbeater.cc:503] Master 127.10.250.126:38593 was elected leader, sending a full tablet report...
I20260620 18:41:36.816685 24486 tablet_bootstrap.cc:492] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e: Bootstrap starting.
I20260620 18:41:36.817668 24486 tablet_bootstrap.cc:654] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:36.818102 24486 log.cc:824] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:36.819485 24486 tablet_bootstrap.cc:492] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e: No bootstrap required, opened a new log
I20260620 18:41:36.819562 24486 ts_tablet_manager.cc:1397] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:41:36.822204 24486 raft_consensus.cc:348] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea188831866c4b768fcbdb44af1bf97e" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44497 } } peers { permanent_uuid: "7ac2bc8f2d1b41eba6803cef2b20b659" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 39937 } } peers { permanent_uuid: "ba82114bbb18450c87506b2d76dc592d" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 35595 } }
I20260620 18:41:36.822399 24486 raft_consensus.cc:374] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:36.822436 24486 raft_consensus.cc:729] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea188831866c4b768fcbdb44af1bf97e, State: Initialized, Role: FOLLOWER
I20260620 18:41:36.822592 24486 consensus_queue.cc:260] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea188831866c4b768fcbdb44af1bf97e" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44497 } } peers { permanent_uuid: "7ac2bc8f2d1b41eba6803cef2b20b659" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 39937 } } peers { permanent_uuid: "ba82114bbb18450c87506b2d76dc592d" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 35595 } }
I20260620 18:41:36.822877 24486 ts_tablet_manager.cc:1428] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:36.823017 24313 heartbeater.cc:503] Master 127.10.250.126:38593 was elected leader, sending a full tablet report...
I20260620 18:41:36.946718 24489 raft_consensus.cc:482] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:36.946877 24489 raft_consensus.cc:504] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea188831866c4b768fcbdb44af1bf97e" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44497 } } peers { permanent_uuid: "7ac2bc8f2d1b41eba6803cef2b20b659" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 39937 } } peers { permanent_uuid: "ba82114bbb18450c87506b2d76dc592d" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 35595 } }
I20260620 18:41:36.947333 24489 leader_election.cc:288] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ac2bc8f2d1b41eba6803cef2b20b659 (127.10.250.67:39937), ba82114bbb18450c87506b2d76dc592d (127.10.250.65:35595)
I20260620 18:41:36.961880 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.948238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:50365 (local address 127.10.250.67:39937)
0620 18:41:36.948302 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:36.948304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:36.954130 (+ 5826us) server_negotiation.cc:407] Connection header received
0620 18:41:36.954634 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:36.954636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:36.954646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:36.954676 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:36.957267 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.957973 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.959361 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.960105 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.960161 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:36.960172 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:36.961691 (+ 1519us) server_negotiation.cc:299] Negotiation successful
0620 18:41:36.961767 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 18:41:36.962527 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.948295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:41883 (local address 127.10.250.65:35595)
0620 18:41:36.948351 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:36.948353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:36.954243 (+ 5890us) server_negotiation.cc:407] Connection header received
0620 18:41:36.954363 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:36.954364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:36.954376 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:36.954408 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:36.955037 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.955782 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.960332 (+ 4550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.961091 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.961146 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:36.961155 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:36.962336 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0620 18:41:36.962396 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:41:36.963079 24490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.947648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:39937 (local address 127.10.250.66:50365)
0620 18:41:36.954031 (+ 6383us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:36.954047 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:36.954546 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:36.954741 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:36.954742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:36.954744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:36.954841 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:36.954843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.958106 (+ 3263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.958107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:36.959235 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:36.959239 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.961215 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.961217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:36.961305 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:36.961607 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 18:41:36.962865 (+ 1258us) client_negotiation.cc:240] Negotiation successful
0620 18:41:36.962936 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6356,"thread_start_us":121,"threads_started":1}
I20260620 18:41:36.963837 24401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f2c1087d25648b59dfb3aac6a53e43e" candidate_uuid: "ea188831866c4b768fcbdb44af1bf97e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac2bc8f2d1b41eba6803cef2b20b659" is_pre_election: true
I20260620 18:41:36.964061 24401 raft_consensus.cc:2454] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea188831866c4b768fcbdb44af1bf97e in term 0.
I20260620 18:41:36.964448 24200 leader_election.cc:302] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac2bc8f2d1b41eba6803cef2b20b659, ea188831866c4b768fcbdb44af1bf97e; no voters:
I20260620 18:41:36.964632 24489 raft_consensus.cc:2781] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:36.964681 24489 raft_consensus.cc:482] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:36.964711 24489 raft_consensus.cc:3037] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:36.965637 24489 raft_consensus.cc:504] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea188831866c4b768fcbdb44af1bf97e" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44497 } } peers { permanent_uuid: "7ac2bc8f2d1b41eba6803cef2b20b659" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 39937 } } peers { permanent_uuid: "ba82114bbb18450c87506b2d76dc592d" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 35595 } }
I20260620 18:41:36.965806 24489 leader_election.cc:288] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [CANDIDATE]: Term 1 election: Requested vote from peers 7ac2bc8f2d1b41eba6803cef2b20b659 (127.10.250.67:39937), ba82114bbb18450c87506b2d76dc592d (127.10.250.65:35595)
I20260620 18:41:36.966163 24401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f2c1087d25648b59dfb3aac6a53e43e" candidate_uuid: "ea188831866c4b768fcbdb44af1bf97e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac2bc8f2d1b41eba6803cef2b20b659"
I20260620 18:41:36.966241 24401 raft_consensus.cc:3037] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:36.967242 24401 raft_consensus.cc:2454] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea188831866c4b768fcbdb44af1bf97e in term 1.
I20260620 18:41:36.967494 24200 leader_election.cc:302] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac2bc8f2d1b41eba6803cef2b20b659, ea188831866c4b768fcbdb44af1bf97e; no voters:
I20260620 18:41:36.967617 24489 raft_consensus.cc:2781] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:36.967782 24489 raft_consensus.cc:686] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [term 1 LEADER]: Becoming Leader. State: Replica: ea188831866c4b768fcbdb44af1bf97e, State: Running, Role: LEADER
I20260620 18:41:36.967877 24489 consensus_queue.cc:237] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea188831866c4b768fcbdb44af1bf97e" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44497 } } peers { permanent_uuid: "7ac2bc8f2d1b41eba6803cef2b20b659" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 39937 } } peers { permanent_uuid: "ba82114bbb18450c87506b2d76dc592d" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 35595 } }
I20260620 18:41:36.968773 23981 catalog_manager.cc:5495] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e reported cstate change: term changed from 0 to 1, leader changed from <none> to ea188831866c4b768fcbdb44af1bf97e (127.10.250.66). New cstate: current_term: 1 leader_uuid: "ea188831866c4b768fcbdb44af1bf97e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea188831866c4b768fcbdb44af1bf97e" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 44497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac2bc8f2d1b41eba6803cef2b20b659" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 39937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba82114bbb18450c87506b2d76dc592d" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260620 18:41:36.963423 24491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.947927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:35595 (local address 127.10.250.66:41883)
0620 18:41:36.954183 (+ 6256us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:36.954192 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:36.954280 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:36.954499 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:36.954500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:36.954502 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:36.954952 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:36.954954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.955898 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.955899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:36.957148 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:36.957152 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.961416 (+ 4264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.961417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:36.961548 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:36.961553 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:41:36.963277 (+ 1724us) client_negotiation.cc:240] Negotiation successful
0620 18:41:36.963330 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6243,"thread_start_us":76,"threads_started":1}
I20260620 18:41:36.969820 24133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f2c1087d25648b59dfb3aac6a53e43e" candidate_uuid: "ea188831866c4b768fcbdb44af1bf97e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba82114bbb18450c87506b2d76dc592d" is_pre_election: true
I20260620 18:41:36.969985 24133 raft_consensus.cc:2454] T 1f2c1087d25648b59dfb3aac6a53e43e P ba82114bbb18450c87506b2d76dc592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea188831866c4b768fcbdb44af1bf97e in term 0.
I20260620 18:41:36.970091 24132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f2c1087d25648b59dfb3aac6a53e43e" candidate_uuid: "ea188831866c4b768fcbdb44af1bf97e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba82114bbb18450c87506b2d76dc592d"
I20260620 18:41:36.970137 24132 raft_consensus.cc:3037] T 1f2c1087d25648b59dfb3aac6a53e43e P ba82114bbb18450c87506b2d76dc592d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:36.971050 24132 raft_consensus.cc:2454] T 1f2c1087d25648b59dfb3aac6a53e43e P ba82114bbb18450c87506b2d76dc592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea188831866c4b768fcbdb44af1bf97e in term 1.
W20260620 18:41:36.977543 24448 tablet.cc:2369] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 18:41:36 dist-test-slave-wp6g krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980896, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for HTTP/127.10.250.84@KRBTEST.COM
W20260620 18:41:37.000097 24314 tablet.cc:2369] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:37.002322 24476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.998777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37948 (local address 127.10.250.66:44497)
0620 18:41:36.998819 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:36.998821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:36.998828 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:41:36.998871 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:36.998872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:36.998896 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:36.998956 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:36.999133 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.999903 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:37.001155 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:37.001477 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:37.001576 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:37.001803 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:37.001943 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:37.002019 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:37.002200 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 18:41:37.002225 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 18:41:37.006424 24180 tablet.cc:2369] T 1f2c1087d25648b59dfb3aac6a53e43e P ba82114bbb18450c87506b2d76dc592d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:37.007227 24132 raft_consensus.cc:1261] T 1f2c1087d25648b59dfb3aac6a53e43e P ba82114bbb18450c87506b2d76dc592d [term 1 FOLLOWER]: Refusing update from remote peer ea188831866c4b768fcbdb44af1bf97e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:37.007597 24493 consensus_queue.cc:1035] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba82114bbb18450c87506b2d76dc592d" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:37.010200 24401 raft_consensus.cc:1261] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659 [term 1 FOLLOWER]: Refusing update from remote peer ea188831866c4b768fcbdb44af1bf97e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:37.010502 24493 consensus_queue.cc:1035] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac2bc8f2d1b41eba6803cef2b20b659" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 39937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:37.016281 24496 mvcc.cc:205] Tried to move back new op lower bound from 7298993754136371200 to 7298993753981071360. Current Snapshot: MvccSnapshot[applied={T|T < 7298993754136371200}]
I20260620 18:41:37.016479 24499 mvcc.cc:205] Tried to move back new op lower bound from 7298993754136371200 to 7298993753981071360. Current Snapshot: MvccSnapshot[applied={T|T < 7298993754136371200}]
I20260620 18:41:37.019065 24497 mvcc.cc:205] Tried to move back new op lower bound from 7298993754136371200 to 7298993753981071360. Current Snapshot: MvccSnapshot[applied={T|T < 7298993754136371200}]
I20260620 18:41:37.021931 23981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:37.022029 23981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:37.022886 23981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e72add3952c045c0b212a8a75a3a7d2d: Sending DeleteTablet for 3 replicas of tablet 1f2c1087d25648b59dfb3aac6a53e43e
I20260620 18:41:37.023314 24247 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f2c1087d25648b59dfb3aac6a53e43e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:37 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:37934
I20260620 18:41:37.023326 24374 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f2c1087d25648b59dfb3aac6a53e43e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:37 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:34894
I20260620 18:41:37.023741 24113 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f2c1087d25648b59dfb3aac6a53e43e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:37 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:46042
I20260620 18:41:37.024056 24505 tablet_replica.cc:330] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e: stopping tablet replica
I20260620 18:41:37.024174 24505 raft_consensus.cc:2229] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:41:37.024324 24505 raft_consensus.cc:2258] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:37.024462 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 24019
I20260620 18:41:37.024523 24506 tablet_replica.cc:330] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659: stopping tablet replica
I20260620 18:41:37.025029 24506 raft_consensus.cc:2229] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:37.025151 24506 raft_consensus.cc:2258] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:37.025774 24506 ts_tablet_manager.cc:1905] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:37.027895 24506 ts_tablet_manager.cc:1918] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:37.027969 24506 log.cc:1196] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/1f2c1087d25648b59dfb3aac6a53e43e
I20260620 18:41:37.028255 24506 ts_tablet_manager.cc:1939] T 1f2c1087d25648b59dfb3aac6a53e43e P 7ac2bc8f2d1b41eba6803cef2b20b659: Deleting consensus metadata
I20260620 18:41:37.028870 23965 catalog_manager.cc:4841] TS 7ac2bc8f2d1b41eba6803cef2b20b659 (127.10.250.67:39937): tablet 1f2c1087d25648b59dfb3aac6a53e43e (table test-table [id=d69630742eef4e73a2cd0ca6f9f75a57]) successfully deleted
I20260620 18:41:37.029991 24505 ts_tablet_manager.cc:1905] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:37.034983 24505 ts_tablet_manager.cc:1918] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:37.035166 24505 log.cc:1196] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/1f2c1087d25648b59dfb3aac6a53e43e
I20260620 18:41:37.035519 24505 ts_tablet_manager.cc:1939] T 1f2c1087d25648b59dfb3aac6a53e43e P ea188831866c4b768fcbdb44af1bf97e: Deleting consensus metadata
I20260620 18:41:37.036054 23965 catalog_manager.cc:4841] TS ea188831866c4b768fcbdb44af1bf97e (127.10.250.66:44497): tablet 1f2c1087d25648b59dfb3aac6a53e43e (table test-table [id=d69630742eef4e73a2cd0ca6f9f75a57]) successfully deleted
W20260620 18:41:37.036592 23967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:35595 (error 108)
I20260620 18:41:37.037487 24479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:37.037376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:35595 (local address 127.0.0.1:46054)
0620 18:41:37.037424 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:37.037454 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 18:41:37.037689 23967 catalog_manager.cc:4584] TS ba82114bbb18450c87506b2d76dc592d (127.10.250.65:35595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f2c1087d25648b59dfb3aac6a53e43e: Network error: Client connection negotiation failed: client connection to 127.10.250.65:35595: connect: Connection refused (error 111)
I20260620 18:41:37.038008 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 24183
I20260620 18:41:37.044814 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 24316
I20260620 18:41:37.052810 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 23948
I20260620 18:41:37.060858 11241 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 18:41:37.405886 11241 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 18:41:37.405987 11241 mini_ranger.cc:66] Stopping Ranger...
I20260620 18:41:37.751118 11241 mini_ranger.cc:68] Stopped Ranger
2026-06-20 18:41:37.751 UTC [23015] LOG: received smart shutdown request
2026-06-20 18:41:37.754 UTC [23015] LOG: background worker "logical replication launcher" (PID 23024) exited with exit code 1
2026-06-20 18:41:37.755 UTC [23019] LOG: shutting down
2026-06-20 18:41:37.918 UTC [23015] LOG: database system is shut down
2026-06-20T18:41:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50987 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:38 dist-test-slave-wp6g krb5kdc[24518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:38 dist-test-slave-wp6g krb5kdc[24518](info): set up 2 sockets
Jun 20 18:41:38 dist-test-slave-wp6g krb5kdc[24518](info): commencing operation
krb5kdc: starting...
W20260620 18:41:39.082446 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:41:39 dist-test-slave-wp6g krb5kdc[24518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:41:39.214241 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:42915
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:37843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:42915,127.10.250.125:42601,127.10.250.124:41385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:39.299109 24531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:39.299322 24531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:39.299346 24531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:39.300592 24531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:39.300644 24531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:39.300658 24531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:39.300673 24531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:39.300736 24531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:39.302278 24531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:37843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:42915,127.10.250.125:42601,127.10.250.124:41385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:42915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:39.302582 24531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:39.302840 24531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:39.307853 24539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:39.307996 24531 server_base.cc:1034] running on GCE node
W20260620 18:41:39.307868 24537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:39.307868 24536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:39.308441 24531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:39.308719 24531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:39.309908 24531 hybrid_clock.cc:648] HybridClock initialized: now 1781980899309881 us; error 37 us; skew 500 ppm
Jun 20 18:41:39 dist-test-slave-wp6g krb5kdc[24518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:39.312678 24531 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:41:39.313297 24531 webserver.cc:460] Webserver started at http://127.10.250.126:42501/ using document root <none> and password file <none>
I20260620 18:41:39.313568 24531 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:39.313616 24531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:39.313773 24531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:39.314980 24531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "a8950a13560e439aa21499848f29237c"
format_stamp: "Formatted at 2026-06-20 18:41:39 on dist-test-slave-wp6g"
server_key: "74841d00cc27fae104b2a93c33af2f5b"
server_key_iv: "653a21dfd27eb7c5daf74dfc13cb3f13"
server_key_version: "encryptionkey@0"
I20260620 18:41:39.315366 24531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "a8950a13560e439aa21499848f29237c"
format_stamp: "Formatted at 2026-06-20 18:41:39 on dist-test-slave-wp6g"
server_key: "74841d00cc27fae104b2a93c33af2f5b"
server_key_iv: "653a21dfd27eb7c5daf74dfc13cb3f13"
server_key_version: "encryptionkey@0"
I20260620 18:41:39.316799 24531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:39.317637 24546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:39.317821 24531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:41:39.317934 24531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "a8950a13560e439aa21499848f29237c"
format_stamp: "Formatted at 2026-06-20 18:41:39 on dist-test-slave-wp6g"
server_key: "74841d00cc27fae104b2a93c33af2f5b"
server_key_iv: "653a21dfd27eb7c5daf74dfc13cb3f13"
server_key_version: "encryptionkey@0"
I20260620 18:41:39.318015 24531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:39.327898 24531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:39.330189 24531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:39.330432 24531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:39.335289 24531 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:42915
I20260620 18:41:39.335335 24598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:42915 every 8 connection(s)
I20260620 18:41:39.335736 24531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:39.336726 24599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:39.338351 24599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:41:39.339598 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 24531
I20260620 18:41:39.339711 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:41:39.339874 11241 external_mini_cluster.cc:1428] Setting key 5eae372ae60dd0cb2e98831619850571
Jun 20 18:41:39 dist-test-slave-wp6g krb5kdc[24518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.250.125@KRBTEST.COM; defaulting to no policy
I20260620 18:41:39.349246 24601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.339144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.10.250.126:42915)
0620 18:41:39.339407 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:39.339448 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:39.339533 (+ 85us) server_negotiation.cc:407] Connection header received
0620 18:41:39.340264 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:39.340266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:39.340283 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:39.340385 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:39.340985 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.341814 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.342885 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.343212 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.343348 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.345598 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:39.345605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:39.345608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:39.345620 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:39.347453 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.347897 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.347900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.347901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.348017 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.348275 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.348276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.348276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.348512 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:39.348610 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:39.348797 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 18:41:39.348946 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":100,"threads_started":1}
I20260620 18:41:39.349247 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.339128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:42915 (local address 127.0.0.1:54350)
0620 18:41:39.339396 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.339445 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:39.340091 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:39.340402 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:39.340405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:39.340712 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:39.340877 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.340881 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.341934 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.341935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.342790 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.342794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.343320 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.343321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.343388 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.344000 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:39.344009 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:39.345484 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:39.347563 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.347570 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.347572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.347793 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.348106 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.348107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.348107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.348206 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.348587 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:39.348591 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:39.348656 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 18:41:39.348770 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 18:41:39.349010 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":87,"thread_start_us":133,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.125@KRBTEST.COM" created.
I20260620 18:41:39.350351 24599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:41:39.350867 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.350687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:42601 (local address 127.0.0.1:38730)
0620 18:41:39.350750 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.350828 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:42601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 18:41:39.351063 24547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.250.125:42601: connect: Connection refused (error 111)
I20260620 18:41:39.351547 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.351435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:42601 (local address 127.0.0.1:38736)
0620 18:41:39.351484 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.351515 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:42601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 18:41:39.351683 24599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.125:42601: Network error: Client connection negotiation failed: client connection to 127.10.250.125:42601: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.125.keytab.
Entry for principal kudu/127.10.250.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.125.keytab.
I20260620 18:41:39.366973 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.125:42601
--webserver_interface=127.10.250.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.125.keytab
--principal=kudu/127.10.250.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:37843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:42915,127.10.250.125:42601,127.10.250.124:41385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:39.400928 24599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } attempt: 1
I20260620 18:41:39.401528 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.401364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:42601 (local address 127.0.0.1:38750)
0620 18:41:39.401443 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.401487 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:42601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 18:41:39.402061 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.401920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:42601 (local address 127.0.0.1:38754)
0620 18:41:39.401997 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.402034 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:42601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 18:41:39.402230 24599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.125:42601: Network error: Client connection negotiation failed: client connection to 127.10.250.125:42601: connect: Connection refused (error 111)
W20260620 18:41:39.453205 24606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:39.453420 24606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:39.453444 24606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:39.454636 24606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:39.454684 24606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:39.454699 24606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:39.454712 24606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:39.454725 24606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:39.456259 24606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:37843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:42915,127.10.250.125:42601,127.10.250.124:41385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.125.keytab
--principal=kudu/127.10.250.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.125:42601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.250.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:39.456530 24606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:39.456785 24606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:39.461841 24612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:39.461895 24606 server_base.cc:1034] running on GCE node
W20260620 18:41:39.461869 24614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:39.462026 24611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:39.462330 24606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:39.462733 24606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:39.463912 24606 hybrid_clock.cc:648] HybridClock initialized: now 1781980899463889 us; error 37 us; skew 500 ppm
Jun 20 18:41:39 dist-test-slave-wp6g krb5kdc[24518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:39.466636 24606 init.cc:377] Logged in from keytab as kudu/127.10.250.125@KRBTEST.COM (short username kudu)
I20260620 18:41:39.467177 24606 webserver.cc:460] Webserver started at http://127.10.250.125:42483/ using document root <none> and password file <none>
I20260620 18:41:39.467439 24606 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:39.467511 24606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:39.467671 24606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:39.468798 24606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/data/instance:
uuid: "a491043ecebe40faaf4f41283e4e075c"
format_stamp: "Formatted at 2026-06-20 18:41:39 on dist-test-slave-wp6g"
server_key: "d90d415f9bf4a36da981ea995827c449"
server_key_iv: "62405795ad2c6c82e8d9aae340854ad8"
server_key_version: "encryptionkey@0"
I20260620 18:41:39.469161 24606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/wal/instance:
uuid: "a491043ecebe40faaf4f41283e4e075c"
format_stamp: "Formatted at 2026-06-20 18:41:39 on dist-test-slave-wp6g"
server_key: "d90d415f9bf4a36da981ea995827c449"
server_key_iv: "62405795ad2c6c82e8d9aae340854ad8"
server_key_version: "encryptionkey@0"
I20260620 18:41:39.470413 24599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } attempt: 2
I20260620 18:41:39.470662 24606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:41:39.470901 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.470729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:42601 (local address 127.0.0.1:38768)
0620 18:41:39.470828 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.470873 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:42601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 18:41:39.471247 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.471164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:42601 (local address 127.0.0.1:38780)
0620 18:41:39.471208 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.471232 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:42601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 18:41:39.471397 24599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.125:42601: Network error: Client connection negotiation failed: client connection to 127.10.250.125:42601: connect: Connection refused (error 111)
I20260620 18:41:39.471635 24621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:39.471814 24606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:39.471937 24606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/wal
uuid: "a491043ecebe40faaf4f41283e4e075c"
format_stamp: "Formatted at 2026-06-20 18:41:39 on dist-test-slave-wp6g"
server_key: "d90d415f9bf4a36da981ea995827c449"
server_key_iv: "62405795ad2c6c82e8d9aae340854ad8"
server_key_version: "encryptionkey@0"
I20260620 18:41:39.472029 24606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:39.489552 24606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:39.491925 24606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:39.492129 24606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:39.496156 24606 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.125:42601
I20260620 18:41:39.496201 24673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.125:42601 every 8 connection(s)
I20260620 18:41:39.496555 24606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/data/info.pb
I20260620 18:41:39.497421 24674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:39.499104 24674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:41:39.502611 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 24606
I20260620 18:41:39.502700 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-1/wal/instance
I20260620 18:41:39.502852 11241 external_mini_cluster.cc:1428] Setting key f3276b75b1de894783abc0b3720dee63
Jun 20 18:41:39 dist-test-slave-wp6g krb5kdc[24518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.125@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:41:39.509287 24601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.499869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.10.250.126:42915)
0620 18:41:39.499980 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:39.499982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:39.500160 (+ 178us) server_negotiation.cc:407] Connection header received
0620 18:41:39.500665 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:39.500666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:39.500679 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:39.500721 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:39.501350 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.501853 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.503202 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.503416 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.503484 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.506053 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:39.506054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:39.506055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:39.506062 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:39.507841 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.508291 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.508292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.508293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.508362 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.508618 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.508619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.508620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.508747 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:39.508826 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:39.509095 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 18:41:39.509185 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:41:39.509491 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.499828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:42915 (local address 127.0.0.1:54360)
0620 18:41:39.500039 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.500076 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:39.500556 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:39.500802 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:39.500806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:39.501062 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:39.501254 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.501258 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.501954 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.501955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.503114 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.503117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.503512 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.503513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.503669 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.504279 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:39.504289 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:39.505956 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:39.507925 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.507932 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.507934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.508196 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.508432 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.508433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.508434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.508550 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.508817 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:39.508823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:39.508961 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 18:41:39.509075 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 18:41:39.509244 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
I20260620 18:41:39.510075 24674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.250.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.124@KRBTEST.COM" created.
Jun 20 18:41:39 dist-test-slave-wp6g krb5kdc[24518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.125@KRBTEST.COM for kudu/127.10.250.125@KRBTEST.COM
I20260620 18:41:39.517745 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.510381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:42601 (local address 127.0.0.1:38782)
0620 18:41:39.510431 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.510441 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:39.510477 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:39.510794 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:39.510795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:39.510939 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:39.510982 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.510984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.511912 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.511913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.512463 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.512465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.512943 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.512944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.513009 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.513464 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:39.513469 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:39.514788 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:39.516624 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.516625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.516625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.516862 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.517092 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.517092 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.517092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.517147 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.517492 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:39.517493 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:39.517531 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:41:39.517584 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 18:41:39.517668 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 18:41:39.517922 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.510458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.10.250.125:42601)
0620 18:41:39.510664 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:39.510668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:39.510683 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:41:39.510698 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:39.510700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:39.510715 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:39.510794 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:39.511057 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.511827 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.512546 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.512857 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.512945 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.514873 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:39.514883 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:39.514883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:39.514891 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:39.516556 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.516958 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.516962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.516962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.517023 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.517215 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.517216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.517217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.517423 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:39.517508 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:39.517617 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 18:41:39.517765 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":184,"thread_start_us":150,"threads_started":1}
I20260620 18:41:39.518667 24674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:41:39.519088 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.518947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41385 (local address 127.0.0.1:59330)
0620 18:41:39.519002 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.519059 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 18:41:39.519276 24623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41385: connect: Connection refused (error 111)
I20260620 18:41:39.519843 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.519723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41385 (local address 127.0.0.1:59344)
0620 18:41:39.519778 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.519812 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 18:41:39.519970 24674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.124:41385: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41385: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.124.keytab.
Entry for principal kudu/127.10.250.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.124.keytab.
I20260620 18:41:39.533329 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.124:41385
--webserver_interface=127.10.250.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.124.keytab
--principal=kudu/127.10.250.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:37843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:42915,127.10.250.125:42601,127.10.250.124:41385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:39.562595 24599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } attempt: 3
Jun 20 18:41:39 dist-test-slave-wp6g krb5kdc[24518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for kudu/127.10.250.125@KRBTEST.COM
I20260620 18:41:39.569222 24674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } attempt: 1
I20260620 18:41:39.569610 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.569505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41385 (local address 127.0.0.1:59350)
0620 18:41:39.569546 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.569581 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 18:41:39.570077 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.569984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41385 (local address 127.0.0.1:59366)
0620 18:41:39.570029 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.570056 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 18:41:39.570228 24674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.124:41385: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41385: connect: Connection refused (error 111)
I20260620 18:41:39.570429 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.563025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:42601 (local address 127.0.0.1:38792)
0620 18:41:39.563082 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.563099 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:39.563165 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:39.563346 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:39.563347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:39.563515 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:39.563583 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.563585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.564324 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.564325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.564837 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.564838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.565214 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.565215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.565295 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.565826 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:39.565832 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:39.567347 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:39.569104 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.569105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.569106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.569304 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.569535 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.569536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.569537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.569618 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.570050 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:39.570052 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:39.570109 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 18:41:39.570187 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:41:39.570325 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 18:41:39.570613 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.563101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.10.250.125:42601)
0620 18:41:39.563210 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:39.563212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:39.563222 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:41:39.563258 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:39.563259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:39.563270 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:39.563338 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:39.563708 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.564227 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.564924 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.565138 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.565216 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.567479 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:39.567483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:39.567483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:39.567496 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:39.568981 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.569386 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.569387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.569388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.569453 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.569708 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.569709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.569709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.569854 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:39.569923 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:39.570350 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 18:41:39.570445 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:41:39.571059 24599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:41:39.571424 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.571324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41385 (local address 127.0.0.1:59374)
0620 18:41:39.571373 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.571399 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 18:41:39.571816 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.571721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41385 (local address 127.0.0.1:59386)
0620 18:41:39.571769 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.571792 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 18:41:39.571929 24599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.124:41385: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41385: connect: Connection refused (error 111)
I20260620 18:41:39.603101 24599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } attempt: 1
I20260620 18:41:39.603658 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.603513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41385 (local address 127.0.0.1:59390)
0620 18:41:39.603588 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.603630 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:41:39.604167 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.604053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41385 (local address 127.0.0.1:59396)
0620 18:41:39.604109 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.604142 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 18:41:39.604319 24599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.124:41385: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41385: connect: Connection refused (error 111)
W20260620 18:41:39.621628 24681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:39.621883 24681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:39.621924 24681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:39.623318 24681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:39.623390 24681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:39.623415 24681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:39.623445 24681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:39.623473 24681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:39.625211 24681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:37843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:42915,127.10.250.125:42601,127.10.250.124:41385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.124.keytab
--principal=kudu/127.10.250.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.124:41385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.250.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:39.625558 24681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:39.625810 24681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:39.630946 24689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:39.630944 24686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:39.630944 24687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:39.631132 24681 server_base.cc:1034] running on GCE node
I20260620 18:41:39.631508 24681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:39.631825 24681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:39.632984 24681 hybrid_clock.cc:648] HybridClock initialized: now 1781980899632974 us; error 27 us; skew 500 ppm
Jun 20 18:41:39 dist-test-slave-wp6g krb5kdc[24518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:39.635674 24681 init.cc:377] Logged in from keytab as kudu/127.10.250.124@KRBTEST.COM (short username kudu)
I20260620 18:41:39.636097 24681 webserver.cc:460] Webserver started at http://127.10.250.124:40925/ using document root <none> and password file <none>
I20260620 18:41:39.636343 24681 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:39.636391 24681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:39.636545 24681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:39.637665 24681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/data/instance:
uuid: "6868c211a37e46a78d3146d7a549755a"
format_stamp: "Formatted at 2026-06-20 18:41:39 on dist-test-slave-wp6g"
server_key: "e44195c05c61f44533c9839f1a0927ac"
server_key_iv: "e77e02721ff930c326ffa840a7722cf2"
server_key_version: "encryptionkey@0"
I20260620 18:41:39.638088 24681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/wal/instance:
uuid: "6868c211a37e46a78d3146d7a549755a"
format_stamp: "Formatted at 2026-06-20 18:41:39 on dist-test-slave-wp6g"
server_key: "e44195c05c61f44533c9839f1a0927ac"
server_key_iv: "e77e02721ff930c326ffa840a7722cf2"
server_key_version: "encryptionkey@0"
I20260620 18:41:39.638379 24674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } attempt: 2
I20260620 18:41:39.638839 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.638693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41385 (local address 127.0.0.1:59402)
0620 18:41:39.638765 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.638799 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 18:41:39.639153 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.639061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41385 (local address 127.0.0.1:59418)
0620 18:41:39.639106 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.639130 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 18:41:39.639249 24674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.124:41385: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41385: connect: Connection refused (error 111)
I20260620 18:41:39.639717 24681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:41:39.640524 24696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:39.640822 24681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:39.640929 24681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/wal
uuid: "6868c211a37e46a78d3146d7a549755a"
format_stamp: "Formatted at 2026-06-20 18:41:39 on dist-test-slave-wp6g"
server_key: "e44195c05c61f44533c9839f1a0927ac"
server_key_iv: "e77e02721ff930c326ffa840a7722cf2"
server_key_version: "encryptionkey@0"
I20260620 18:41:39.641004 24681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:39.659301 24681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:39.661619 24681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:39.661804 24681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:39.665934 24681 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.124:41385
I20260620 18:41:39.665990 24748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.124:41385 every 8 connection(s)
I20260620 18:41:39.666390 24681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/data/info.pb
I20260620 18:41:39.667423 24749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:39.668577 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 24681
I20260620 18:41:39.668678 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980738687848-11241-0/minicluster-data/master-2/wal/instance
I20260620 18:41:39.668844 11241 external_mini_cluster.cc:1428] Setting key ce6bbfea764bde6f19e3a9b530230d86
I20260620 18:41:39.669184 24749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:41:39 dist-test-slave-wp6g krb5kdc[24518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.124@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
Jun 20 18:41:39 dist-test-slave-wp6g krb5kdc[24518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.124@KRBTEST.COM
I20260620 18:41:39.679450 24599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } attempt: 2
I20260620 18:41:39.679628 24601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.669981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.10.250.126:42915)
0620 18:41:39.670054 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:39.670056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:39.670282 (+ 226us) server_negotiation.cc:407] Connection header received
0620 18:41:39.670357 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:39.670358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:39.670370 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:39.670448 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:39.671108 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.671590 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.672661 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.672969 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.673048 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.676101 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:39.676103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:39.676103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:39.676132 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:39.677817 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.678396 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.678398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.678398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.678452 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.678731 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.678731 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.678732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.678849 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:39.678914 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:39.679429 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 18:41:39.679522 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:41:39.679817 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.669923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:42915 (local address 127.0.0.1:54376)
0620 18:41:39.670108 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.670152 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:39.670246 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:39.670483 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:39.670485 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:39.670815 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:39.670978 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.670982 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.671671 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.671672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.672564 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.672566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.673074 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.673075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.673237 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.673911 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:39.673923 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:39.675956 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:39.677906 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.677912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.677915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.678306 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.678498 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.678499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.678499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.678659 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.679075 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:39.679082 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:39.679272 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 18:41:39.679362 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 18:41:39.679539 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":106,"threads_started":1}
I20260620 18:41:39.680334 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.669397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.10.250.124:41385)
0620 18:41:39.669605 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:39.669613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:39.669639 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:41:39.670136 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:39.670144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:39.670218 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:39.670340 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:39.671060 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.672013 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.673057 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.673529 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.673614 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.676526 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:39.676536 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:39.676537 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:39.676546 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:39.678389 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.678792 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.678795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.678795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.678850 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.679065 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.679066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.679067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.679276 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:39.680095 (+ 819us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:39.680153 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 18:41:39.680237 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":93,"threads_started":1}
I20260620 18:41:39.680460 24749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:41:39 dist-test-slave-wp6g krb5kdc[24518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for kudu/127.10.250.124@KRBTEST.COM
Jun 20 18:41:39 dist-test-slave-wp6g krb5kdc[24518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.124@KRBTEST.COM for kudu/127.10.250.125@KRBTEST.COM
Jun 20 18:41:39 dist-test-slave-wp6g krb5kdc[24518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:41:39.688277 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.680692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:42601 (local address 127.0.0.1:38798)
0620 18:41:39.680774 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.680783 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:39.680816 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:39.681007 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:39.681008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:39.681160 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:39.681195 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.681197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.681911 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.681912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.682723 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.682726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.683158 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.683159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.683206 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.683598 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:39.683603 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:39.685202 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:39.687145 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.687146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.687147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.687375 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.687620 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.687621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.687622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.687691 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.687990 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:39.687990 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:39.688023 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:41:39.688076 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 18:41:39.688151 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:41:39.688350 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.680827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38798 (local address 127.10.250.125:42601)
0620 18:41:39.680898 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:39.680900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:39.680910 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:41:39.680935 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:39.680936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:39.680946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:39.680997 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:39.681280 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.681833 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.682805 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.683084 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.683156 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.685308 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:39.685310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:39.685310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:39.685319 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:39.687046 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.687469 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.687472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.687472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.687534 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.687777 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.687778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.687778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.687917 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:39.687952 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:39.688143 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 18:41:39.688235 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:41:39.688830 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.679721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41385 (local address 127.0.0.1:59428)
0620 18:41:39.679805 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.679818 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:39.679855 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:39.680297 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:39.680298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:39.680449 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:39.680495 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.680497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.681486 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.681487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.682208 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.682211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.682633 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.682634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.682690 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.683089 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:39.683095 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:39.684668 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:39.687394 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.687395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.687395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.687621 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.687839 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.687840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.687840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.688156 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.688467 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:39.688468 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:39.688516 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 18:41:39.688607 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 18:41:39.688731 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 18:41:39.688844 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.679924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59428 (local address 127.10.250.124:41385)
0620 18:41:39.680099 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:39.680101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:39.680108 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:41:39.680134 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:39.680135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:39.680145 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:39.680218 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:39.680578 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.681396 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.682300 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.682565 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.682633 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.684772 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:39.684774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:39.684774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:39.684780 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:39.687300 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.687707 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.687707 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.687708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.687761 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.688236 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.688237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.688237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.688374 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:39.688468 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:39.688632 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 18:41:39.688713 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":118,"threads_started":1}
I20260620 18:41:39.689242 24749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:41:39.690580 24601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.681210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54380 (local address 127.10.250.126:42915)
0620 18:41:39.681244 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:39.681246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:39.681268 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:41:39.681341 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:39.681342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:39.681350 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:39.681399 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:39.681988 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.682722 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.684018 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.684220 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.684295 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.687060 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:39.687062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:39.687062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:39.687069 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:39.689389 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.689788 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.689789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.689790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.689845 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.689994 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.689994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.689995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.690111 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:39.690334 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:39.690407 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 18:41:39.690477 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:41:39.690732 24599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c: Bootstrap starting.
I20260620 18:41:39.691416 24599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:39.691720 24599 log.cc:824] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:39.692584 24599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c: No bootstrap required, opened a new log
Jun 20 18:41:39 dist-test-slave-wp6g krb5kdc[24518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.124@KRBTEST.COM for kudu/127.10.250.124@KRBTEST.COM
I20260620 18:41:39.695387 24599 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } }
I20260620 18:41:39.695549 24599 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:39.695587 24599 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8950a13560e439aa21499848f29237c, State: Initialized, Role: FOLLOWER
I20260620 18:41:39.695729 24599 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } }
I20260620 18:41:39.696105 24599 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:39.696051 24756 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } } }
I20260620 18:41:39.696161 24756 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 18:41:39 dist-test-slave-wp6g krb5kdc[24518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.125@KRBTEST.COM
I20260620 18:41:39.699111 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.689540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41385 (local address 127.0.0.1:59436)
0620 18:41:39.692004 (+ 2464us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.692015 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:39.692084 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:39.692195 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:39.692196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:39.692327 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:39.692354 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.692355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.693047 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.693048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.693522 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.693524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.693916 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.693917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.693965 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.694418 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:39.694423 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:39.695770 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:39.697894 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.697895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.697895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.698139 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.698375 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.698376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.698377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.698431 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.698823 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:39.698824 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:39.698860 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:41:39.698907 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 18:41:39.699019 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2449}
W20260620 18:41:39.699368 24767 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:41:39.699421 24767 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:41:39.699427 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.689596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.10.250.124:41385)
0620 18:41:39.689632 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:39.689634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:39.692070 (+ 2436us) server_negotiation.cc:407] Connection header received
0620 18:41:39.692142 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:39.692143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:39.692152 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:39.692195 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:39.692435 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.692971 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.693615 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.693833 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.693917 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.695974 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:39.695976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:39.695976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:39.695984 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:39.697798 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.698230 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.698231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.698232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.698299 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.698505 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.698506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.698506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.698747 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:39.699220 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:39.699261 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 18:41:39.699351 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 18:41:39.700205 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.691477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.10.250.125:42601)
0620 18:41:39.691517 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:39.691518 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:39.691526 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:41:39.691546 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:39.691547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:39.691555 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:39.691609 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:39.692212 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.692891 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.693840 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.694119 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.694163 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.696892 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:39.696894 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:39.696895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:39.696903 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:39.698428 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.698848 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.698850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.698850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.698911 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.699117 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.699117 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.699118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.699259 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:39.699440 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:39.700037 (+ 597us) server_negotiation.cc:299] Negotiation successful
0620 18:41:39.700115 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:41:39.700889 24749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a: Bootstrap starting.
I20260620 18:41:39.701514 24749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:39.701808 24749 log.cc:824] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:39.702695 24749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a: No bootstrap required, opened a new log
I20260620 18:41:39.704082 24749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } }
I20260620 18:41:39.704223 24749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:39.704272 24749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6868c211a37e46a78d3146d7a549755a, State: Initialized, Role: FOLLOWER
I20260620 18:41:39.704536 24749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } }
I20260620 18:41:39.702383 24531 master_runner.cc:186] Error getting master registration for 127.10.250.125:42601: OK, instance_id { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" instance_seqno: 1781980899494934 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:41:39.704823 24531 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:41:39.705062 24749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:39.705638 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:41:39.706650 24770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } } }
I20260620 18:41:39.706751 24770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [sys.catalog]: This master's current role is: FOLLOWER
W20260620 18:41:39.707410 24781 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:41:39.707466 24781 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:41:39.708650 24681 master_runner.cc:186] Error getting master registration for 127.10.250.125:42601: OK, instance_id { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" instance_seqno: 1781980899494934 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:41:39.711040 24681 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:41:39.713522 24770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:39.713619 24770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } }
I20260620 18:41:39.713881 24770 leader_election.cc:288] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8950a13560e439aa21499848f29237c (127.10.250.126:42915), a491043ecebe40faaf4f41283e4e075c (127.10.250.125:42601)
I20260620 18:41:39.714228 24573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6868c211a37e46a78d3146d7a549755a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8950a13560e439aa21499848f29237c" is_pre_election: true
I20260620 18:41:39.714807 24573 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6868c211a37e46a78d3146d7a549755a in term 0.
I20260620 18:41:39.715039 24700 leader_election.cc:302] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6868c211a37e46a78d3146d7a549755a, a8950a13560e439aa21499848f29237c; no voters:
I20260620 18:41:39.715276 24770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:39.715335 24770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:39.715363 24770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:39.716189 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.707415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.10.250.124:41385)
0620 18:41:39.707478 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:39.707481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:39.707908 (+ 427us) server_negotiation.cc:407] Connection header received
0620 18:41:39.708015 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:39.708016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:39.708028 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:39.708058 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:39.708828 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.709519 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.710548 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.710837 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.710881 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.712653 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:39.712654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:39.712655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:39.712662 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:39.714814 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.715291 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.715293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.715293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.715353 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.715570 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.715571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.715571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.715699 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:39.715794 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:39.716005 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 18:41:39.716098 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:41:39.716195 24601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.707372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54386 (local address 127.10.250.126:42915)
0620 18:41:39.707467 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:39.707469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:39.707776 (+ 307us) server_negotiation.cc:407] Connection header received
0620 18:41:39.707876 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:39.707877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:39.707887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:39.707917 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:39.708651 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.709402 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.710436 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.710753 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.710799 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.712677 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:39.712679 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:39.712679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:39.712687 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:39.714490 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.715258 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.715259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.715260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.715331 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.715610 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.715611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.715611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.715737 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:39.715805 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:39.716028 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 18:41:39.716108 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:41:39.716361 24770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } }
I20260620 18:41:39.716481 24770 leader_election.cc:288] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [CANDIDATE]: Term 1 election: Requested vote from peers a8950a13560e439aa21499848f29237c (127.10.250.126:42915), a491043ecebe40faaf4f41283e4e075c (127.10.250.125:42601)
I20260620 18:41:39.716789 24573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6868c211a37e46a78d3146d7a549755a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8950a13560e439aa21499848f29237c"
I20260620 18:41:39.716898 24573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:39.718076 24573 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6868c211a37e46a78d3146d7a549755a in term 1.
I20260620 18:41:39.718581 24700 leader_election.cc:302] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6868c211a37e46a78d3146d7a549755a, a8950a13560e439aa21499848f29237c; no voters:
I20260620 18:41:39.718667 24770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:39.718839 24770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [term 1 LEADER]: Becoming Leader. State: Replica: 6868c211a37e46a78d3146d7a549755a, State: Running, Role: LEADER
I20260620 18:41:39.718941 24770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } }
I20260620 18:41:39.719352 24770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6868c211a37e46a78d3146d7a549755a. Latest consensus state: current_term: 1 leader_uuid: "6868c211a37e46a78d3146d7a549755a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } } }
I20260620 18:41:39.719424 24770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:39.722719 24792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:39.722951 24792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:39.725248 24573 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c [term 1 FOLLOWER]: Refusing update from remote peer 6868c211a37e46a78d3146d7a549755a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:39.725697 24790 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:39.727334 24756 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6868c211a37e46a78d3146d7a549755a. Latest consensus state: current_term: 1 leader_uuid: "6868c211a37e46a78d3146d7a549755a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } } }
I20260620 18:41:39.727421 24756 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:39.729050 24790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6868c211a37e46a78d3146d7a549755a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } } }
I20260620 18:41:39.729144 24790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:39.729387 24756 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6868c211a37e46a78d3146d7a549755a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } } }
I20260620 18:41:39.729468 24756 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:39.729835 24793 mvcc.cc:205] Tried to move back new op lower bound from 7298993765270831104 to 7298993765249540096. Current Snapshot: MvccSnapshot[applied={T|T < 7298993765270831104}]
I20260620 18:41:39.730108 24794 mvcc.cc:205] Tried to move back new op lower bound from 7298993765270831104 to 7298993765249540096. Current Snapshot: MvccSnapshot[applied={T|T < 7298993765270831104 or (T in {7298993765270831104})}]
I20260620 18:41:39.730393 24674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } attempt: 3
Jun 20 18:41:39 dist-test-slave-wp6g krb5kdc[24518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.125@KRBTEST.COM for kudu/127.10.250.124@KRBTEST.COM
I20260620 18:41:39.735424 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.707955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.10.250.125:42601)
0620 18:41:39.708002 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:39.708003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:39.713414 (+ 5411us) server_negotiation.cc:407] Connection header received
0620 18:41:39.713892 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:39.713894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:39.713907 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:39.713940 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:39.716375 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.717220 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.718232 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.718547 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.718612 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.731889 (+ 13277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:39.731890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:39.731891 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:39.731898 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:39.733787 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.734199 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.734200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.734200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.734269 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.734501 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.734502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.734502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.734667 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:39.734779 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:39.735243 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 18:41:39.735323 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 18:41:39.736228 24792 catalog_manager.cc:1347] Generated new cluster ID: 569e9c22647a45a79ec1fbd4c2d1ad2f
I20260620 18:41:39.736279 24792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:39.739207 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.730733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41385 (local address 127.0.0.1:59456)
0620 18:41:39.730792 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.730806 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:39.730859 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:39.731013 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:39.731014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:39.731209 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:39.731277 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.731279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.732225 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.732226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.733039 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:39.733041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.733536 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.733537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:39.733613 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.734251 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:39.734287 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:39.736027 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:39.737937 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.737938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.737938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.738164 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.738438 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:39.738439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:39.738439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:39.738536 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:39.738878 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:39.738880 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:39.738940 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 18:41:39.739002 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 18:41:39.739101 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 18:41:39.740072 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.730786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.10.250.124:41385)
0620 18:41:39.730851 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:39.730853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:39.730864 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:41:39.730922 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:39.730923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:39.730934 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:39.730987 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:39.731359 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.732124 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.733135 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:39.733437 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:39.733498 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:39.736151 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:39.736152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:39.736153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:39.736178 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:39.737816 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.738276 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.738294 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.738295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.738350 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:39.738626 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:39.738627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:39.738628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:39.738774 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:39.738831 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:39.739654 (+ 823us) server_negotiation.cc:299] Negotiation successful
0620 18:41:39.739843 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:41:39.742116 24674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c: Bootstrap starting.
I20260620 18:41:39.742954 24674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:39.743279 24674 log.cc:824] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:39.755409 24674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c: No bootstrap required, opened a new log
I20260620 18:41:39.757728 24674 raft_consensus.cc:348] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } }
I20260620 18:41:39.758577 24674 raft_consensus.cc:374] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:39.758667 24674 raft_consensus.cc:729] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a491043ecebe40faaf4f41283e4e075c, State: Initialized, Role: FOLLOWER
I20260620 18:41:39.758839 24674 consensus_queue.cc:260] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } }
I20260620 18:41:39.759243 24674 sys_catalog.cc:564] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:39.759306 24647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6868c211a37e46a78d3146d7a549755a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a491043ecebe40faaf4f41283e4e075c"
I20260620 18:41:39.759446 24647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:39.760432 24647 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6868c211a37e46a78d3146d7a549755a in term 1.
I20260620 18:41:39.760964 24646 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c [term 1 FOLLOWER]: Refusing update from remote peer 6868c211a37e46a78d3146d7a549755a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:39.760711 24799 sys_catalog.cc:455] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } } }
I20260620 18:41:39.761101 24799 sys_catalog.cc:458] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:39.761487 24770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:39.762125 24648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6868c211a37e46a78d3146d7a549755a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a491043ecebe40faaf4f41283e4e075c" is_pre_election: true
I20260620 18:41:39.762213 24648 raft_consensus.cc:2362] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6868c211a37e46a78d3146d7a549755a in term 1. Re-sending same reply.
I20260620 18:41:39.763958 24799 sys_catalog.cc:455] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6868c211a37e46a78d3146d7a549755a. Latest consensus state: current_term: 1 leader_uuid: "6868c211a37e46a78d3146d7a549755a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } } }
I20260620 18:41:39.764048 24799 sys_catalog.cc:458] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:39.764097 24799 sys_catalog.cc:455] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6868c211a37e46a78d3146d7a549755a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } } }
I20260620 18:41:39.764138 24799 sys_catalog.cc:458] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:39.765237 24792 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:39.766002 24792 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:39.766846 24770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6868c211a37e46a78d3146d7a549755a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8950a13560e439aa21499848f29237c" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42915 } } peers { permanent_uuid: "a491043ecebe40faaf4f41283e4e075c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 42601 } } peers { permanent_uuid: "6868c211a37e46a78d3146d7a549755a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41385 } } }
I20260620 18:41:39.766956 24770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:39.771271 24810 catalog_manager.cc:1259] Loaded cluster ID: 569e9c22647a45a79ec1fbd4c2d1ad2f
I20260620 18:41:39.771330 24810 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c: loading cluster ID for follower catalog manager: success
I20260620 18:41:39.773451 24792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6868c211a37e46a78d3146d7a549755a: Generated new TSK 0
I20260620 18:41:39.773650 24792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:41:39.773944 24810 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c: acquiring CA information for follower catalog manager: success
W20260620 18:41:39.774063 24810 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260620 18:41:40.699885 24767 catalog_manager.cc:1259] Loaded cluster ID: 569e9c22647a45a79ec1fbd4c2d1ad2f
I20260620 18:41:40.699963 24767 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c: loading cluster ID for follower catalog manager: success
I20260620 18:41:40.700778 24767 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c: acquiring CA information for follower catalog manager: success
I20260620 18:41:40.701043 24767 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a8950a13560e439aa21499848f29237c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:41:40.774825 24810 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a491043ecebe40faaf4f41283e4e075c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:41:40.828614 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 24531
I20260620 18:41:40.833873 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 24606
I20260620 18:41:40.839079 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 24681
2026-06-20T18:41:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2809 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:40 dist-test-slave-wp6g krb5kdc[24822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:40 dist-test-slave-wp6g krb5kdc[24822](info): set up 2 sockets
Jun 20 18:41:40 dist-test-slave-wp6g krb5kdc[24822](info): commencing operation
krb5kdc: starting...
W20260620 18:41:41.890779 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:41:41 dist-test-slave-wp6g krb5kdc[24822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:41:42.017818 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:33851
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:36283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:33851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:42.101315 24835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:42.101545 24835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:42.101569 24835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:42.102779 24835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:42.102823 24835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:42.102837 24835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:42.102850 24835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:42.102862 24835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:42.104305 24835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:33851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:33851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:42.104503 24835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:42.104682 24835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:42.110085 24841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:42.110076 24840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:42.110219 24835 server_base.cc:1034] running on GCE node
W20260620 18:41:42.110195 24843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:42.110757 24835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:42.111164 24835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:42.112350 24835 hybrid_clock.cc:648] HybridClock initialized: now 1781980902112334 us; error 36 us; skew 500 ppm
Jun 20 18:41:42 dist-test-slave-wp6g krb5kdc[24822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980902, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:42.115093 24835 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:41:42.115685 24835 webserver.cc:460] Webserver started at http://127.10.250.126:40367/ using document root <none> and password file <none>
I20260620 18:41:42.115945 24835 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:42.116017 24835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:42.116183 24835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:42.117318 24835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "8f9d17384efc44a098985baeba9c340d"
format_stamp: "Formatted at 2026-06-20 18:41:42 on dist-test-slave-wp6g"
server_key: "9c476edf10a343dfef6688613ebe4ecf"
server_key_iv: "5b27489055ff7d1d6e219914b159ac78"
server_key_version: "encryptionkey@0"
I20260620 18:41:42.117702 24835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "8f9d17384efc44a098985baeba9c340d"
format_stamp: "Formatted at 2026-06-20 18:41:42 on dist-test-slave-wp6g"
server_key: "9c476edf10a343dfef6688613ebe4ecf"
server_key_iv: "5b27489055ff7d1d6e219914b159ac78"
server_key_version: "encryptionkey@0"
I20260620 18:41:42.119261 24835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:41:42.120077 24850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:42.120375 24835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:42.120514 24835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "8f9d17384efc44a098985baeba9c340d"
format_stamp: "Formatted at 2026-06-20 18:41:42 on dist-test-slave-wp6g"
server_key: "9c476edf10a343dfef6688613ebe4ecf"
server_key_iv: "5b27489055ff7d1d6e219914b159ac78"
server_key_version: "encryptionkey@0"
I20260620 18:41:42.120604 24835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:42.166923 24835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:42.169080 24835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:42.169210 24835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:42.173687 24835 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:33851
I20260620 18:41:42.173760 24902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:33851 every 8 connection(s)
I20260620 18:41:42.174139 24835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:42.175164 24903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:42.177515 24903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d: Bootstrap starting.
I20260620 18:41:42.178141 24903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:42.178442 24903 log.cc:824] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:42.179234 24903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d: No bootstrap required, opened a new log
I20260620 18:41:42.180994 24903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9d17384efc44a098985baeba9c340d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 33851 } }
I20260620 18:41:42.181183 24903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:42.181209 24903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9d17384efc44a098985baeba9c340d, State: Initialized, Role: FOLLOWER
I20260620 18:41:42.181391 24903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9d17384efc44a098985baeba9c340d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 33851 } }
I20260620 18:41:42.181463 24903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:42.181486 24903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:42.181517 24903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:42.182245 24903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9d17384efc44a098985baeba9c340d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 33851 } }
I20260620 18:41:42.182348 24903 leader_election.cc:302] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f9d17384efc44a098985baeba9c340d; no voters:
I20260620 18:41:42.182613 24903 leader_election.cc:288] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:42.182709 24906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:42.182904 24906 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [term 1 LEADER]: Becoming Leader. State: Replica: 8f9d17384efc44a098985baeba9c340d, State: Running, Role: LEADER
I20260620 18:41:42.182938 24903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:42.183038 24906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9d17384efc44a098985baeba9c340d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 33851 } }
I20260620 18:41:42.183674 24907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f9d17384efc44a098985baeba9c340d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9d17384efc44a098985baeba9c340d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 33851 } } }
I20260620 18:41:42.183789 24907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:42.184067 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 24835
I20260620 18:41:42.184156 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:41:42.184185 24915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:42.184154 24908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f9d17384efc44a098985baeba9c340d. Latest consensus state: current_term: 1 leader_uuid: "8f9d17384efc44a098985baeba9c340d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9d17384efc44a098985baeba9c340d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 33851 } } }
I20260620 18:41:42.184247 24908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:42.184397 11241 external_mini_cluster.cc:1428] Setting key b66d44f53a8969f5c54ca24b149464e5
I20260620 18:41:42.184769 24915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:42.186792 24915 catalog_manager.cc:1347] Generated new cluster ID: d0722c8693394a1f8d2699832dd4a344
I20260620 18:41:42.186851 24915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:41:42 dist-test-slave-wp6g krb5kdc[24822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:41:42.195274 24925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:42.184864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.10.250.126:33851)
0620 18:41:42.185056 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:42.185069 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:42.185254 (+ 185us) server_negotiation.cc:407] Connection header received
0620 18:41:42.185440 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:42.185448 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:42.185519 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:42.185684 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:42.186328 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:42.187440 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:42.188199 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:42.188658 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:42.188786 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:42.191278 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:42.191282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:42.191283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:42.191301 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:42.193573 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:42.193969 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:42.193972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:42.193973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:42.194085 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:42.194325 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:42.194326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:42.194326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:42.194648 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:42.194824 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:42.194927 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 18:41:42.195087 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
I20260620 18:41:42.196211 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:41:42.200311 24915 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:42.201049 24915 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:42.209733 24915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f9d17384efc44a098985baeba9c340d: Generated new TSK 0
I20260620 18:41:42.209992 24915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:41:42.250152 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 24835
2026-06-20T18:41:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1409 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:42 dist-test-slave-wp6g krb5kdc[24930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:42 dist-test-slave-wp6g krb5kdc[24930](info): set up 2 sockets
Jun 20 18:41:42 dist-test-slave-wp6g krb5kdc[24930](info): commencing operation
krb5kdc: starting...
W20260620 18:41:43.306794 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:41:43 dist-test-slave-wp6g krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:41:43.429364 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:41709
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:41709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:43.513907 24943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:43.514142 24943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:43.514168 24943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:43.515419 24943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:43.515463 24943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:43.515478 24943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:43.515491 24943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:43.515503 24943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:43.516989 24943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:41709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:41709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:43.517215 24943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:43.517416 24943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:43.522387 24949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:43.522404 24948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:43.522492 24943 server_base.cc:1034] running on GCE node
W20260620 18:41:43.522387 24951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:43.522934 24943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:43.523205 24943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:43.524333 24943 hybrid_clock.cc:648] HybridClock initialized: now 1781980903524313 us; error 42 us; skew 500 ppm
Jun 20 18:41:43 dist-test-slave-wp6g krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980903, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:43.531116 24943 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:41:43.531648 24943 webserver.cc:460] Webserver started at http://127.10.250.126:42583/ using document root <none> and password file <none>
I20260620 18:41:43.531925 24943 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:43.531975 24943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:43.532164 24943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:43.533366 24943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "ba8e17c78f2e4468a94d06ab18a312aa"
format_stamp: "Formatted at 2026-06-20 18:41:43 on dist-test-slave-wp6g"
server_key: "af67454624bd8c6e84a9fb15ad381d43"
server_key_iv: "220e416b7ba12fa587ec5d09256a0dde"
server_key_version: "encryptionkey@0"
I20260620 18:41:43.533751 24943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "ba8e17c78f2e4468a94d06ab18a312aa"
format_stamp: "Formatted at 2026-06-20 18:41:43 on dist-test-slave-wp6g"
server_key: "af67454624bd8c6e84a9fb15ad381d43"
server_key_iv: "220e416b7ba12fa587ec5d09256a0dde"
server_key_version: "encryptionkey@0"
I20260620 18:41:43.535300 24943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:41:43.536206 24958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:43.536464 24943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:43.536542 24943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "ba8e17c78f2e4468a94d06ab18a312aa"
format_stamp: "Formatted at 2026-06-20 18:41:43 on dist-test-slave-wp6g"
server_key: "af67454624bd8c6e84a9fb15ad381d43"
server_key_iv: "220e416b7ba12fa587ec5d09256a0dde"
server_key_version: "encryptionkey@0"
I20260620 18:41:43.536646 24943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:43.548720 24943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:43.551105 24943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:43.551261 24943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:43.555595 24943 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:41709
I20260620 18:41:43.555631 25010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:41709 every 8 connection(s)
I20260620 18:41:43.556036 24943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:43.557031 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:43.559491 25011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa: Bootstrap starting.
I20260620 18:41:43.560113 25011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:43.560408 25011 log.cc:824] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:43.561184 25011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa: No bootstrap required, opened a new log
I20260620 18:41:43.562955 25011 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } }
I20260620 18:41:43.563104 25011 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:43.563127 25011 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8e17c78f2e4468a94d06ab18a312aa, State: Initialized, Role: FOLLOWER
I20260620 18:41:43.563309 25011 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } }
I20260620 18:41:43.563439 25011 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:43.563467 25011 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:43.563520 25011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:43.564451 25011 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } }
I20260620 18:41:43.564582 25011 leader_election.cc:302] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba8e17c78f2e4468a94d06ab18a312aa; no voters:
I20260620 18:41:43.564637 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 24943
I20260620 18:41:43.564738 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:41:43.564814 25011 leader_election.cc:288] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:43.564877 25014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:43.564953 11241 external_mini_cluster.cc:1428] Setting key 854d6f6c0e97a644ae83d13f87123769
I20260620 18:41:43.565038 25014 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 1 LEADER]: Becoming Leader. State: Replica: ba8e17c78f2e4468a94d06ab18a312aa, State: Running, Role: LEADER
I20260620 18:41:43.565100 25011 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:43.565141 25014 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } }
I20260620 18:41:43.565500 25014 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba8e17c78f2e4468a94d06ab18a312aa. Latest consensus state: current_term: 1 leader_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } } }
I20260620 18:41:43.565623 25014 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:43.565501 25015 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } } }
I20260620 18:41:43.565788 25015 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:43.566104 25021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:43.567536 25021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:43.569561 25021 catalog_manager.cc:1347] Generated new cluster ID: d6d67effd18a4cdcbb4be72224cdb4ae
I20260620 18:41:43.569620 25021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:41:43 dist-test-slave-wp6g krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:41:43.576936 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:43.565684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.10.250.126:41709)
0620 18:41:43.565872 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:43.565883 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:43.566055 (+ 172us) server_negotiation.cc:407] Connection header received
0620 18:41:43.566367 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:43.566376 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:43.566441 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:43.566618 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:43.567721 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:43.569083 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:43.569990 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:43.570434 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:43.570616 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:43.573032 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:43.573037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:43.573037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:43.573050 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:43.575279 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:43.575674 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:43.575678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:43.575678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:43.575795 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:43.576010 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:43.576011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:43.576012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:43.576285 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:43.576452 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:43.576560 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 18:41:43.576712 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260620 18:41:43.583114 25021 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:43.583768 25021 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:43.592806 25021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa: Generated new TSK 0
I20260620 18:41:43.593117 25021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:41:43.623509 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:41709
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:43.713232 25037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:43.713493 25037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:43.713536 25037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:43.714766 25037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:43.714836 25037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:43.714884 25037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:41:43.716395 25037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:43.716703 25037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:43.716943 25037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:41:43.722386 25037 server_base.cc:1034] running on GCE node
W20260620 18:41:43.722417 25042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:43.722335 25045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:43.722330 25043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:43.722808 25037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:43.723152 25037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:43.724346 25037 hybrid_clock.cc:648] HybridClock initialized: now 1781980903724325 us; error 37 us; skew 500 ppm
Jun 20 18:41:43 dist-test-slave-wp6g krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980903, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:43.727119 25037 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:41:43.727557 25037 webserver.cc:460] Webserver started at http://127.10.250.65:38285/ using document root <none> and password file <none>
I20260620 18:41:43.727746 25037 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:43.727787 25037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:43.727877 25037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:43.728885 25037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "5784328fa9b04541acf2c1a5a1cdc9f0"
format_stamp: "Formatted at 2026-06-20 18:41:43 on dist-test-slave-wp6g"
server_key: "b82508a5b5e3fcb194b630637b71d766"
server_key_iv: "6a6cc96ede1d7def88eef8d2f719c228"
server_key_version: "encryptionkey@0"
I20260620 18:41:43.729266 25037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "5784328fa9b04541acf2c1a5a1cdc9f0"
format_stamp: "Formatted at 2026-06-20 18:41:43 on dist-test-slave-wp6g"
server_key: "b82508a5b5e3fcb194b630637b71d766"
server_key_iv: "6a6cc96ede1d7def88eef8d2f719c228"
server_key_version: "encryptionkey@0"
I20260620 18:41:43.730676 25037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:41:43.731408 25052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:43.731595 25037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:43.731685 25037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "5784328fa9b04541acf2c1a5a1cdc9f0"
format_stamp: "Formatted at 2026-06-20 18:41:43 on dist-test-slave-wp6g"
server_key: "b82508a5b5e3fcb194b630637b71d766"
server_key_iv: "6a6cc96ede1d7def88eef8d2f719c228"
server_key_version: "encryptionkey@0"
I20260620 18:41:43.731756 25037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:43.742061 25037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:43.744163 25037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:43.744310 25037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:43.744496 25037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:43.744815 25037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:43.744848 25037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:43.744894 25037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:43.744920 25037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:43.750593 25037 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:39303
I20260620 18:41:43.750610 25165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:39303 every 8 connection(s)
I20260620 18:41:43.750978 25037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:41:43 dist-test-slave-wp6g krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980903, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:41:43.759500 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 25037
I20260620 18:41:43.759601 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:41:43.759788 11241 external_mini_cluster.cc:1428] Setting key 920f228f9fc9d69bbe9c1a49515bfd4c
I20260620 18:41:43.761862 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:43.751806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:53381 (local address 127.10.250.126:41709)
0620 18:41:43.751871 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:43.751873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:43.752038 (+ 165us) server_negotiation.cc:407] Connection header received
0620 18:41:43.752571 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:43.752572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:43.752583 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:43.752648 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:43.753658 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:43.754187 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:43.755555 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:43.755896 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:43.755978 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:43.758781 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:43.758783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:43.758784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:43.758791 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:43.760306 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:43.760824 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:43.760825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:43.760825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:43.760890 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:43.761134 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:43.761135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:43.761136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:43.761297 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:43.761367 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:43.761609 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 18:41:43.761721 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:41:43.761883 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:43.751753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:53381)
0620 18:41:43.751913 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:43.751957 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:43.752454 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:43.752748 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:43.752751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:43.753349 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:43.753556 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:43.753561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:43.754283 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:43.754284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:43.755447 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:43.755450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:43.756007 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:43.756008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:43.756183 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:43.756907 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:43.756920 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:43.758688 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:43.760387 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:43.760395 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:43.760397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:43.760737 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:43.760960 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:43.760960 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:43.760960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:43.761065 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:43.761357 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:43.761360 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:43.761484 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 18:41:43.761565 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 18:41:43.761691 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260620 18:41:43.762562 25166 heartbeater.cc:347] Connected to a master server at 127.10.250.126:41709
I20260620 18:41:43.762688 25166 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:43.762876 25166 heartbeater.cc:511] Master 127.10.250.126:41709 requested a full tablet report, sending...
I20260620 18:41:43.763458 24975 ts_manager.cc:194] Registered new tserver with Master: 5784328fa9b04541acf2c1a5a1cdc9f0 (127.10.250.65:39303)
I20260620 18:41:43.764091 24975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:53381
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:41:43.791462 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:41709
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:43.874576 25171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:43.874796 25171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:43.874821 25171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:43.876098 25171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:43.876144 25171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:43.876173 25171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:41:43.877641 25171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:43.877837 25171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:43.878021 25171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:43.883348 25179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:43.883374 25177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:43.883503 25171 server_base.cc:1034] running on GCE node
W20260620 18:41:43.883348 25176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:43.883816 25171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:43.884027 25171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:43.885198 25171 hybrid_clock.cc:648] HybridClock initialized: now 1781980903885227 us; error 74 us; skew 500 ppm
Jun 20 18:41:43 dist-test-slave-wp6g krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980903, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:43.888360 25171 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:41:43.888909 25171 webserver.cc:460] Webserver started at http://127.10.250.66:41295/ using document root <none> and password file <none>
I20260620 18:41:43.889169 25171 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:43.889222 25171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:43.889385 25171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:43.890604 25171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "d00b58e8de1847619ef01f4fefc700e4"
format_stamp: "Formatted at 2026-06-20 18:41:43 on dist-test-slave-wp6g"
server_key: "007269bf68602c27cbb5cc64bd19a2b6"
server_key_iv: "833d1207f833ca04e47e80b1af24c12c"
server_key_version: "encryptionkey@0"
I20260620 18:41:43.891007 25171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "d00b58e8de1847619ef01f4fefc700e4"
format_stamp: "Formatted at 2026-06-20 18:41:43 on dist-test-slave-wp6g"
server_key: "007269bf68602c27cbb5cc64bd19a2b6"
server_key_iv: "833d1207f833ca04e47e80b1af24c12c"
server_key_version: "encryptionkey@0"
I20260620 18:41:43.892552 25171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:41:43.893570 25186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:43.893796 25171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:43.893913 25171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "d00b58e8de1847619ef01f4fefc700e4"
format_stamp: "Formatted at 2026-06-20 18:41:43 on dist-test-slave-wp6g"
server_key: "007269bf68602c27cbb5cc64bd19a2b6"
server_key_iv: "833d1207f833ca04e47e80b1af24c12c"
server_key_version: "encryptionkey@0"
I20260620 18:41:43.893996 25171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:43.908496 25171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:43.911100 25171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:43.911291 25171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:43.911547 25171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:43.911921 25171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:43.911980 25171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:43.912021 25171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:43.912074 25171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:43.918290 25171 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:37833
I20260620 18:41:43.918406 25299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:37833 every 8 connection(s)
I20260620 18:41:43.918732 25171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:41:43 dist-test-slave-wp6g krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980903, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:41:43.926771 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 25171
I20260620 18:41:43.926898 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:41:43.927114 11241 external_mini_cluster.cc:1428] Setting key 2a584395424a060de19fe64e9733889c
I20260620 18:41:43.928988 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:43.919567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:52753 (local address 127.10.250.126:41709)
0620 18:41:43.919655 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:43.919658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:43.919861 (+ 203us) server_negotiation.cc:407] Connection header received
0620 18:41:43.920404 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:43.920405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:43.920415 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:43.920475 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:43.921433 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:43.921931 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:43.923012 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:43.923245 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:43.923310 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:43.925699 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:43.925701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:43.925701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:43.925708 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:43.927478 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:43.927945 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:43.927946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:43.927946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:43.928015 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:43.928295 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:43.928296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:43.928296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:43.928423 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:43.928500 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:43.928802 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 18:41:43.928894 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:41:43.929209 25302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:43.919521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:52753)
0620 18:41:43.919726 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:43.919764 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:43.920288 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:43.920533 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:43.920536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:43.921127 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:43.921328 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:43.921334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:43.922018 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:43.922019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:43.922933 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:43.922935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:43.923324 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:43.923325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:43.923445 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:43.924014 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:43.924023 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:43.925603 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:43.927566 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:43.927576 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:43.927578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:43.927872 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:43.928078 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:43.928079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:43.928079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:43.928219 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:43.928492 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:43.928499 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:43.928666 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 18:41:43.928782 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:41:43.928955 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":111,"threads_started":1}
I20260620 18:41:43.929719 25300 heartbeater.cc:347] Connected to a master server at 127.10.250.126:41709
I20260620 18:41:43.929837 25300 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:43.930051 25300 heartbeater.cc:511] Master 127.10.250.126:41709 requested a full tablet report, sending...
I20260620 18:41:43.930505 24975 ts_manager.cc:194] Registered new tserver with Master: d00b58e8de1847619ef01f4fefc700e4 (127.10.250.66:37833)
I20260620 18:41:43.930944 24975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:52753
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:41:43.954108 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:41709
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:44.038671 25305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:44.038915 25305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:44.038959 25305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:44.040237 25305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:44.040305 25305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:44.040352 25305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:41:44.042263 25305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:44.042608 25305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:44.042886 25305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:44.048554 25311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:44.048547 25313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:44.048547 25310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:44.049069 25305 server_base.cc:1034] running on GCE node
I20260620 18:41:44.049346 25305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:44.049652 25305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:44.050808 25305 hybrid_clock.cc:648] HybridClock initialized: now 1781980904050790 us; error 37 us; skew 500 ppm
Jun 20 18:41:44 dist-test-slave-wp6g krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980904, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:44.053561 25305 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:41:44.054049 25305 webserver.cc:460] Webserver started at http://127.10.250.67:40545/ using document root <none> and password file <none>
I20260620 18:41:44.054291 25305 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:44.054361 25305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:44.054572 25305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:44.055707 25305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "1959ea8b02e34e2290cc0e5a0ce27a98"
format_stamp: "Formatted at 2026-06-20 18:41:44 on dist-test-slave-wp6g"
server_key: "49edb73759d44aaba5edc784cdc78096"
server_key_iv: "b550a70a78e4e022902d3eeb1972d2f9"
server_key_version: "encryptionkey@0"
I20260620 18:41:44.056075 25305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "1959ea8b02e34e2290cc0e5a0ce27a98"
format_stamp: "Formatted at 2026-06-20 18:41:44 on dist-test-slave-wp6g"
server_key: "49edb73759d44aaba5edc784cdc78096"
server_key_iv: "b550a70a78e4e022902d3eeb1972d2f9"
server_key_version: "encryptionkey@0"
I20260620 18:41:44.057590 25305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:41:44.058578 25320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:44.058789 25305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:41:44.058916 25305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "1959ea8b02e34e2290cc0e5a0ce27a98"
format_stamp: "Formatted at 2026-06-20 18:41:44 on dist-test-slave-wp6g"
server_key: "49edb73759d44aaba5edc784cdc78096"
server_key_iv: "b550a70a78e4e022902d3eeb1972d2f9"
server_key_version: "encryptionkey@0"
I20260620 18:41:44.058995 25305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:44.082329 25305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:44.085563 25305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:44.085701 25305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:44.085917 25305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:44.086632 25305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:44.086679 25305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:44.086705 25305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:44.086719 25305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:44.093964 25305 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:33425
I20260620 18:41:44.094127 25433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:33425 every 8 connection(s)
I20260620 18:41:44.094398 25305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:41:44.099640 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 25305
I20260620 18:41:44.099742 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:41:44.099948 11241 external_mini_cluster.cc:1428] Setting key 63c79d1d73fe60818fc7edaee7edaabc
Jun 20 18:41:44 dist-test-slave-wp6g krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980904, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:41:44.105588 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.095460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:41651 (local address 127.10.250.126:41709)
0620 18:41:44.095523 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.095527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.095705 (+ 178us) server_negotiation.cc:407] Connection header received
0620 18:41:44.096268 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.096269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.096284 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.096365 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:44.097479 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.098024 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.099181 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.099379 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.099450 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.102343 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:44.102345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:44.102345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:44.102353 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:44.103998 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.104496 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.104498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.104498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.104567 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.104809 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.104810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.104810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.104929 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:44.104981 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.105432 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.105519 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 18:41:44.105800 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.095376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:41651)
0620 18:41:44.095541 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.095585 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:44.096111 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:44.096479 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:44.096484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:44.097151 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:44.097383 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.097388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.098135 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.098135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.099074 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.099077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.099496 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.099497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.099676 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.100474 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:44.100488 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:44.102239 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:44.104124 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:44.104132 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:44.104134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:44.104402 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:44.104641 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:44.104642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:44.104642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:44.104750 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:44.104991 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:44.104997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:44.105140 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 18:41:44.105378 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 18:41:44.105538 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260620 18:41:44.106556 25434 heartbeater.cc:347] Connected to a master server at 127.10.250.126:41709
I20260620 18:41:44.106669 25434 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:44.106835 25434 heartbeater.cc:511] Master 127.10.250.126:41709 requested a full tablet report, sending...
I20260620 18:41:44.107249 24975 ts_manager.cc:194] Registered new tserver with Master: 1959ea8b02e34e2290cc0e5a0ce27a98 (127.10.250.67:33425)
I20260620 18:41:44.107646 24975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:41651
I20260620 18:41:44.112380 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:41:44.120519 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.113826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60676 (local address 127.10.250.126:41709)
0620 18:41:44.113934 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.113936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.114037 (+ 101us) server_negotiation.cc:407] Connection header received
0620 18:41:44.114135 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.114136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.114147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.114234 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:44.114942 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.115444 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.116169 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.116371 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.116445 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.117764 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:44.117765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:44.117766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:44.117774 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:44.119308 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.119674 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.119675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.119675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.119739 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.119946 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.119947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.119947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.120108 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:44.120226 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.120326 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.120413 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:41:44.122363 24975 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:41:44.124081 24975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:41:44.130980 25446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.126712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:39303 (local address 127.0.0.1:42644)
0620 18:41:44.127074 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.127084 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:44.127159 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:44.127365 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:44.127368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:44.127370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:44.127449 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.127452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.128809 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.128810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.129879 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.129881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.130627 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.130629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.130729 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.130746 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 18:41:44.130818 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:41:44.130881 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":89,"threads_started":1}
I20260620 18:41:44.130980 25445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.126658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:33425 (local address 127.0.0.1:35472)
0620 18:41:44.126845 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.126890 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:44.126958 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:44.127517 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:44.127517 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:44.127519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:44.127550 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.127551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.128508 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.128509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.129785 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.129788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.130782 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.130783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.130843 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.130846 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:41:44.130895 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 18:41:44.130925 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":77,"threads_started":1}
I20260620 18:41:44.131060 25447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.126781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.10.250.67:33425)
0620 18:41:44.127282 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.127287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.127297 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:41:44.127339 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.127343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.127360 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.127436 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:44.127633 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.128398 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.129897 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.130700 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.130783 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.130846 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.130910 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.130959 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":86,"threads_started":1}
I20260620 18:41:44.131692 25448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.126824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.10.250.65:39303)
0620 18:41:44.127023 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.127029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.127184 (+ 155us) server_negotiation.cc:407] Connection header received
0620 18:41:44.127241 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.127244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.127263 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.127356 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:44.127526 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.128692 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.129942 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.130466 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.130627 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.130701 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.131542 (+ 841us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.131605 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":92,"threads_started":1}
I20260620 18:41:44.131707 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.126919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:37833 (local address 127.0.0.1:57400)
0620 18:41:44.127089 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.127101 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:44.127167 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:44.127693 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:44.127694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:44.127696 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:44.127744 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.127746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.129095 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.129096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.129857 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.129860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.131439 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.131440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.131506 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.131512 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:41:44.131579 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 18:41:44.131629 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
I20260620 18:41:44.131740 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.127155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57400 (local address 127.10.250.66:37833)
0620 18:41:44.127415 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.127431 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.127442 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:41:44.127484 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.127486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.127503 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.127827 (+ 324us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:44.127901 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.128957 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.129970 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.131336 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.131439 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.131506 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.131594 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.131642 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260620 18:41:44.132195 25368 tablet_service.cc:1467] Processing CreateTablet for tablet 55aaa1917a9d45ae9b0564ed4200fc9b (DEFAULT_TABLE table=test-table [id=fc3214b2a253453684d4d56efb494a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:44.132431 25100 tablet_service.cc:1467] Processing CreateTablet for tablet 55aaa1917a9d45ae9b0564ed4200fc9b (DEFAULT_TABLE table=test-table [id=fc3214b2a253453684d4d56efb494a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:44.132581 25368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55aaa1917a9d45ae9b0564ed4200fc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:44.132735 25100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55aaa1917a9d45ae9b0564ed4200fc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:44.132683 25234 tablet_service.cc:1467] Processing CreateTablet for tablet 55aaa1917a9d45ae9b0564ed4200fc9b (DEFAULT_TABLE table=test-table [id=fc3214b2a253453684d4d56efb494a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:44.132974 25234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55aaa1917a9d45ae9b0564ed4200fc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:44.136253 25451 tablet_bootstrap.cc:492] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0: Bootstrap starting.
I20260620 18:41:44.136313 25452 tablet_bootstrap.cc:492] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98: Bootstrap starting.
I20260620 18:41:44.136924 25452 tablet_bootstrap.cc:654] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:44.136927 25451 tablet_bootstrap.cc:654] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:44.137416 25452 log.cc:824] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:44.137449 25451 log.cc:824] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:44.137794 25453 tablet_bootstrap.cc:492] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4: Bootstrap starting.
I20260620 18:41:44.138301 25451 tablet_bootstrap.cc:492] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0: No bootstrap required, opened a new log
I20260620 18:41:44.138310 25452 tablet_bootstrap.cc:492] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98: No bootstrap required, opened a new log
I20260620 18:41:44.138382 25452 ts_tablet_manager.cc:1397] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:41:44.138382 25451 ts_tablet_manager.cc:1397] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:41:44.138403 25453 tablet_bootstrap.cc:654] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:44.138898 25453 log.cc:824] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:44.139776 25453 tablet_bootstrap.cc:492] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4: No bootstrap required, opened a new log
I20260620 18:41:44.139868 25453 ts_tablet_manager.cc:1397] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:41:44.140738 25452 raft_consensus.cc:348] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } }
I20260620 18:41:44.140738 25451 raft_consensus.cc:348] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } }
I20260620 18:41:44.140935 25451 raft_consensus.cc:374] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:44.140934 25452 raft_consensus.cc:374] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:44.140976 25451 raft_consensus.cc:729] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5784328fa9b04541acf2c1a5a1cdc9f0, State: Initialized, Role: FOLLOWER
I20260620 18:41:44.140995 25452 raft_consensus.cc:729] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1959ea8b02e34e2290cc0e5a0ce27a98, State: Initialized, Role: FOLLOWER
I20260620 18:41:44.141137 25451 consensus_queue.cc:260] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } }
I20260620 18:41:44.141136 25452 consensus_queue.cc:260] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } }
I20260620 18:41:44.141469 25452 ts_tablet_manager.cc:1428] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:44.141496 25166 heartbeater.cc:503] Master 127.10.250.126:41709 was elected leader, sending a full tablet report...
I20260620 18:41:44.141595 25434 heartbeater.cc:503] Master 127.10.250.126:41709 was elected leader, sending a full tablet report...
I20260620 18:41:44.141800 25451 ts_tablet_manager.cc:1428] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:44.141896 25453 raft_consensus.cc:348] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } }
I20260620 18:41:44.142078 25453 raft_consensus.cc:374] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:44.142118 25453 raft_consensus.cc:729] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00b58e8de1847619ef01f4fefc700e4, State: Initialized, Role: FOLLOWER
I20260620 18:41:44.142232 25453 consensus_queue.cc:260] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } }
I20260620 18:41:44.142554 25453 ts_tablet_manager.cc:1428] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:44.142609 25300 heartbeater.cc:503] Master 127.10.250.126:41709 was elected leader, sending a full tablet report...
I20260620 18:41:44.163679 25459 raft_consensus.cc:482] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:44.163852 25459 raft_consensus.cc:504] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } }
I20260620 18:41:44.164351 25459 leader_election.cc:288] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5784328fa9b04541acf2c1a5a1cdc9f0 (127.10.250.65:39303), 1959ea8b02e34e2290cc0e5a0ce27a98 (127.10.250.67:33425)
I20260620 18:41:44.168808 25447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.164611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:35935 (local address 127.10.250.67:33425)
0620 18:41:44.164654 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.164656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.164844 (+ 188us) server_negotiation.cc:407] Connection header received
0620 18:41:44.164907 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.164908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.164918 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.164975 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:44.165250 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.166032 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.167762 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.168396 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.168463 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.168474 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.168687 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.168731 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:41:44.168808 25448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.164591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:59697 (local address 127.10.250.65:39303)
0620 18:41:44.164638 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.164641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.164688 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:41:44.164782 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.164783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.164799 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.164877 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:44.165251 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.166022 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.167776 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.168375 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.168464 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.168474 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.168685 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.168731 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:41:44.168810 25302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.164526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:39303 (local address 127.10.250.66:59697)
0620 18:41:44.164619 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.164632 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:44.164701 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:44.164899 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:44.164900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:44.164902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:44.165003 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.165005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.166186 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.166187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.167652 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.167657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.168489 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.168490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.168590 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.168593 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:41:44.168657 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 18:41:44.168708 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":142}
I20260620 18:41:44.168817 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.164526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:33425 (local address 127.10.250.66:35935)
0620 18:41:44.164763 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.164772 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:44.164834 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:44.164994 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:44.164994 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:44.164996 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:44.165065 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.165066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.166186 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.166187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.167666 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.167668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.168489 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.168490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.168584 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.168589 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:41:44.168666 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 18:41:44.168707 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":92,"thread_start_us":118,"threads_started":1}
I20260620 18:41:44.169189 25120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55aaa1917a9d45ae9b0564ed4200fc9b" candidate_uuid: "d00b58e8de1847619ef01f4fefc700e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" is_pre_election: true
I20260620 18:41:44.169261 25388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55aaa1917a9d45ae9b0564ed4200fc9b" candidate_uuid: "d00b58e8de1847619ef01f4fefc700e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" is_pre_election: true
W20260620 18:41:44.169358 25301 tablet.cc:2369] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:44.169420 25388 raft_consensus.cc:2454] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00b58e8de1847619ef01f4fefc700e4 in term 0.
I20260620 18:41:44.169420 25120 raft_consensus.cc:2454] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00b58e8de1847619ef01f4fefc700e4 in term 0.
I20260620 18:41:44.169639 25189 leader_election.cc:302] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5784328fa9b04541acf2c1a5a1cdc9f0, d00b58e8de1847619ef01f4fefc700e4; no voters:
I20260620 18:41:44.169796 25459 raft_consensus.cc:2781] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:44.169859 25459 raft_consensus.cc:482] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:44.169875 25459 raft_consensus.cc:3037] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:44.170737 25459 raft_consensus.cc:504] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } }
I20260620 18:41:44.170888 25459 leader_election.cc:288] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [CANDIDATE]: Term 1 election: Requested vote from peers 5784328fa9b04541acf2c1a5a1cdc9f0 (127.10.250.65:39303), 1959ea8b02e34e2290cc0e5a0ce27a98 (127.10.250.67:33425)
I20260620 18:41:44.171072 25388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55aaa1917a9d45ae9b0564ed4200fc9b" candidate_uuid: "d00b58e8de1847619ef01f4fefc700e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98"
I20260620 18:41:44.171149 25388 raft_consensus.cc:3037] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:44.171113 25120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55aaa1917a9d45ae9b0564ed4200fc9b" candidate_uuid: "d00b58e8de1847619ef01f4fefc700e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0"
I20260620 18:41:44.171236 25120 raft_consensus.cc:3037] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:44.171967 25388 raft_consensus.cc:2454] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00b58e8de1847619ef01f4fefc700e4 in term 1.
I20260620 18:41:44.172070 25120 raft_consensus.cc:2454] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00b58e8de1847619ef01f4fefc700e4 in term 1.
I20260620 18:41:44.172176 25188 leader_election.cc:302] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1959ea8b02e34e2290cc0e5a0ce27a98, d00b58e8de1847619ef01f4fefc700e4; no voters:
I20260620 18:41:44.172307 25459 raft_consensus.cc:2781] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:44.172515 25459 raft_consensus.cc:686] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [term 1 LEADER]: Becoming Leader. State: Replica: d00b58e8de1847619ef01f4fefc700e4, State: Running, Role: LEADER
I20260620 18:41:44.172603 25459 consensus_queue.cc:237] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } }
I20260620 18:41:44.173514 24975 catalog_manager.cc:5495] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d00b58e8de1847619ef01f4fefc700e4 (127.10.250.66). New cstate: current_term: 1 leader_uuid: "d00b58e8de1847619ef01f4fefc700e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } health_report { overall_health: HEALTHY } } }
I20260620 18:41:44.191146 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.188555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.10.250.66:37833)
0620 18:41:44.188637 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.188639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.188649 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:41:44.188694 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.188695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.188718 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.188811 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:44.188964 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.189453 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.190252 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.190468 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.190589 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.190764 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:44.190886 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:44.190991 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.191009 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.191032 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:41:44.193361 25120 raft_consensus.cc:1261] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 1 FOLLOWER]: Refusing update from remote peer d00b58e8de1847619ef01f4fefc700e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:44.193367 25388 raft_consensus.cc:1261] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 1 FOLLOWER]: Refusing update from remote peer d00b58e8de1847619ef01f4fefc700e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:44.193800 25459 consensus_queue.cc:1035] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:44.193986 25461 consensus_queue.cc:1035] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:44.196496 25464 mvcc.cc:205] Tried to move back new op lower bound from 7298993783572930560 to 7298993783491842048. Current Snapshot: MvccSnapshot[applied={T|T < 7298993783572930560}]
I20260620 18:41:44.197733 25465 mvcc.cc:205] Tried to move back new op lower bound from 7298993783572930560 to 7298993783491842048. Current Snapshot: MvccSnapshot[applied={T|T < 7298993783572930560}]
I20260620 18:41:44.199327 24974 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:44.199424 24974 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:44.200289 24974 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa: Sending DeleteTablet for 3 replicas of tablet 55aaa1917a9d45ae9b0564ed4200fc9b
I20260620 18:41:44.200685 25368 tablet_service.cc:1514] Processing DeleteTablet for tablet 55aaa1917a9d45ae9b0564ed4200fc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:44 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:35472
I20260620 18:41:44.200750 25100 tablet_service.cc:1514] Processing DeleteTablet for tablet 55aaa1917a9d45ae9b0564ed4200fc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:44 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:42644
I20260620 18:41:44.200744 25234 tablet_service.cc:1514] Processing DeleteTablet for tablet 55aaa1917a9d45ae9b0564ed4200fc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:44 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:57400
I20260620 18:41:44.200958 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 25037
I20260620 18:41:44.200984 25475 tablet_replica.cc:330] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4: stopping tablet replica
I20260620 18:41:44.201565 25475 raft_consensus.cc:2229] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:41:44.201686 25475 raft_consensus.cc:2258] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:44.202203 25475 ts_tablet_manager.cc:1905] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:44.204748 25475 ts_tablet_manager.cc:1918] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:44.204937 25475 log.cc:1196] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/55aaa1917a9d45ae9b0564ed4200fc9b
I20260620 18:41:44.205338 25475 ts_tablet_manager.cc:1939] T 55aaa1917a9d45ae9b0564ed4200fc9b P d00b58e8de1847619ef01f4fefc700e4: Deleting consensus metadata
I20260620 18:41:44.205999 24961 catalog_manager.cc:4841] TS d00b58e8de1847619ef01f4fefc700e4 (127.10.250.66:37833): tablet 55aaa1917a9d45ae9b0564ed4200fc9b (table test-table [id=fc3214b2a253453684d4d56efb494a24]) successfully deleted
I20260620 18:41:44.206079 25473 tablet_replica.cc:330] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98: stopping tablet replica
I20260620 18:41:44.206218 25473 raft_consensus.cc:2229] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:44.206357 25473 raft_consensus.cc:2258] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:44.206945 25473 ts_tablet_manager.cc:1905] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 18:41:44.207355 24961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:39303 (error 108)
I20260620 18:41:44.207485 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 25171
I20260620 18:41:44.208407 25446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.208253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:39303 (local address 127.0.0.1:42660)
0620 18:41:44.208307 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.208360 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:39303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 18:41:44.208633 24961 catalog_manager.cc:4584] TS 5784328fa9b04541acf2c1a5a1cdc9f0 (127.10.250.65:39303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55aaa1917a9d45ae9b0564ed4200fc9b: Network error: Client connection negotiation failed: client connection to 127.10.250.65:39303: connect: Connection refused (error 111)
I20260620 18:41:44.209486 25473 ts_tablet_manager.cc:1918] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:44.209585 25473 log.cc:1196] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/55aaa1917a9d45ae9b0564ed4200fc9b
I20260620 18:41:44.209872 25473 ts_tablet_manager.cc:1939] T 55aaa1917a9d45ae9b0564ed4200fc9b P 1959ea8b02e34e2290cc0e5a0ce27a98: Deleting consensus metadata
I20260620 18:41:44.210389 24960 catalog_manager.cc:4841] TS 1959ea8b02e34e2290cc0e5a0ce27a98 (127.10.250.67:33425): tablet 55aaa1917a9d45ae9b0564ed4200fc9b (table test-table [id=fc3214b2a253453684d4d56efb494a24]) successfully deleted
I20260620 18:41:44.214350 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 25305
I20260620 18:41:44.219239 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 24943
I20260620 18:41:44.224679 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:41709
--webserver_interface=127.10.250.126
--webserver_port=42583
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:41709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:44.307363 25477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:44.307591 25477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:44.307616 25477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:44.309115 25477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:44.309188 25477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:44.309206 25477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:44.309221 25477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:44.309235 25477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:44.311215 25477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:41709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:41709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=42583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:44.311528 25477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:44.311815 25477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:44.321288 25484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:44.321388 25477 server_base.cc:1034] running on GCE node
W20260620 18:41:44.321301 25483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:44.321316 25486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:44.323030 25477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:44.323480 25477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:44.326856 25477 hybrid_clock.cc:648] HybridClock initialized: now 1781980904326778 us; error 66 us; skew 500 ppm
Jun 20 18:41:44 dist-test-slave-wp6g krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980904, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:44.329942 25477 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:41:44.330500 25477 webserver.cc:460] Webserver started at http://127.10.250.126:42583/ using document root <none> and password file <none>
I20260620 18:41:44.330834 25477 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:44.330893 25477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:44.332507 25477 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:41:44.333655 25493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:44.333967 25477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:41:44.334090 25477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "ba8e17c78f2e4468a94d06ab18a312aa"
format_stamp: "Formatted at 2026-06-20 18:41:43 on dist-test-slave-wp6g"
server_key: "af67454624bd8c6e84a9fb15ad381d43"
server_key_iv: "220e416b7ba12fa587ec5d09256a0dde"
server_key_version: "encryptionkey@0"
I20260620 18:41:44.334620 25477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:44.365411 25477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:44.368021 25477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:44.368234 25477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:44.373721 25477 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:41709
I20260620 18:41:44.373760 25545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:41709 every 8 connection(s)
I20260620 18:41:44.374250 25477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:44.376325 25546 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:41:44.377233 25546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa: Bootstrap starting.
I20260620 18:41:44.380434 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 25477
I20260620 18:41:44.380766 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:39303
--local_ip_for_outbound_sockets=127.10.250.65
--tserver_master_addrs=127.10.250.126:41709
--webserver_port=38285
--webserver_interface=127.10.250.65
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:44.388535 25546 log.cc:824] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:44.391044 25546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:41:44.391410 25546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa: Bootstrap complete.
I20260620 18:41:44.394140 25546 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } }
I20260620 18:41:44.394449 25546 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8e17c78f2e4468a94d06ab18a312aa, State: Initialized, Role: FOLLOWER
I20260620 18:41:44.394662 25546 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } }
I20260620 18:41:44.394737 25546 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:44.394811 25546 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:44.394862 25546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:41:44.396040 25546 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } }
I20260620 18:41:44.396160 25546 leader_election.cc:302] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba8e17c78f2e4468a94d06ab18a312aa; no voters:
I20260620 18:41:44.396368 25546 leader_election.cc:288] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:41:44.396432 25550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:41:44.396587 25550 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 2 LEADER]: Becoming Leader. State: Replica: ba8e17c78f2e4468a94d06ab18a312aa, State: Running, Role: LEADER
I20260620 18:41:44.396689 25546 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:44.396724 25550 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } }
I20260620 18:41:44.397014 25552 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } } }
I20260620 18:41:44.397122 25552 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:44.397013 25550 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba8e17c78f2e4468a94d06ab18a312aa. Latest consensus state: current_term: 2 leader_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } } }
I20260620 18:41:44.397329 25550 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:44.397517 25556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:44.398379 25556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:44.398584 25556 catalog_manager.cc:1259] Loaded cluster ID: d6d67effd18a4cdcbb4be72224cdb4ae
I20260620 18:41:44.398845 25556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:44.399109 25556 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 18:41:44.401793 25556 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 18:41:44.406777 25556 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:44.407023 25556 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa: Loaded TSK: 0
I20260620 18:41:44.407267 25556 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:41:44.479825 25548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:44.480091 25548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:44.480136 25548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:44.481354 25548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:44.481426 25548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:44.481473 25548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:41:44.483136 25548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:39303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=38285
--tserver_master_addrs=127.10.250.126:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:44.483430 25548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:44.483683 25548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:44.489099 25573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:44.489181 25576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:44.489122 25574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:44.489705 25548 server_base.cc:1034] running on GCE node
I20260620 18:41:44.489934 25548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:44.490193 25548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:44.491364 25548 hybrid_clock.cc:648] HybridClock initialized: now 1781980904491338 us; error 42 us; skew 500 ppm
Jun 20 18:41:44 dist-test-slave-wp6g krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980904, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:44.493937 25548 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:41:44.494459 25548 webserver.cc:460] Webserver started at http://127.10.250.65:38285/ using document root <none> and password file <none>
I20260620 18:41:44.494763 25548 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:44.494824 25548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:44.496417 25548 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:41:44.497253 25583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:44.497411 25548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:41:44.497526 25548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "5784328fa9b04541acf2c1a5a1cdc9f0"
format_stamp: "Formatted at 2026-06-20 18:41:43 on dist-test-slave-wp6g"
server_key: "b82508a5b5e3fcb194b630637b71d766"
server_key_iv: "6a6cc96ede1d7def88eef8d2f719c228"
server_key_version: "encryptionkey@0"
I20260620 18:41:44.497905 25548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:44.514747 25548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:44.517174 25548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:44.517386 25548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:44.517711 25548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:44.518213 25590 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 18:41:44.519779 25548 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 18:41:44.519840 25548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 18:41:44.519881 25548 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 18:41:44.520581 25548 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 18:41:44.520627 25548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 18:41:44.520663 25590 tablet_bootstrap.cc:492] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0: Bootstrap starting.
I20260620 18:41:44.528831 25548 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:39303
I20260620 18:41:44.529009 25697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:39303 every 8 connection(s)
I20260620 18:41:44.529412 25548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:41:44.532500 25590 log.cc:824] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:44.535054 25590 tablet_bootstrap.cc:492] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:41:44.535364 25590 tablet_bootstrap.cc:492] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0: Bootstrap complete.
I20260620 18:41:44.536586 25590 ts_tablet_manager.cc:1397] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260620 18:41:44.537770 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 25548
I20260620 18:41:44.538040 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:37833
--local_ip_for_outbound_sockets=127.10.250.66
--tserver_master_addrs=127.10.250.126:41709
--webserver_port=41295
--webserver_interface=127.10.250.66
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:41:44 dist-test-slave-wp6g krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980904, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:41:44.538717 25590 raft_consensus.cc:348] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } }
I20260620 18:41:44.539086 25590 raft_consensus.cc:729] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5784328fa9b04541acf2c1a5a1cdc9f0, State: Initialized, Role: FOLLOWER
I20260620 18:41:44.539215 25590 consensus_queue.cc:260] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } }
I20260620 18:41:44.539465 25590 ts_tablet_manager.cc:1428] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260620 18:41:44.543733 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.530148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:56153)
0620 18:41:44.530648 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.530695 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:44.531294 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:44.531890 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:44.531896 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:44.532313 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:44.532606 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.532614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.534241 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.534242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.535518 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.535521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.536388 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.536390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.536536 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.537188 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:44.537198 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:44.539947 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:44.541808 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:44.541814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:44.541816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:44.542102 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:44.542335 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:44.542336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:44.542336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:44.542465 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:44.543122 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:44.543128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:44.543272 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 18:41:44.543370 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:41:44.543526 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":101,"threads_started":1}
I20260620 18:41:44.544263 25701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.530216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:56153 (local address 127.10.250.126:41709)
0620 18:41:44.530690 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.530705 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.531452 (+ 747us) server_negotiation.cc:407] Connection header received
0620 18:41:44.531588 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.531594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.531658 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.531795 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:44.532771 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.534112 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.535611 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.536291 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.536402 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.540055 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:44.540059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:44.540059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:44.540069 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:44.541723 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.542189 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.542192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.542192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.542276 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.542639 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.542641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.542642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.543034 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:44.543150 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.543512 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.543688 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":82,"threads_started":1}
I20260620 18:41:44.544991 25698 heartbeater.cc:347] Connected to a master server at 127.10.250.126:41709
I20260620 18:41:44.545102 25698 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:44.545329 25698 heartbeater.cc:511] Master 127.10.250.126:41709 requested a full tablet report, sending...
I20260620 18:41:44.546180 25510 ts_manager.cc:194] Registered new tserver with Master: 5784328fa9b04541acf2c1a5a1cdc9f0 (127.10.250.65:39303)
I20260620 18:41:44.547420 25510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:56153
I20260620 18:41:44.548645 25698 heartbeater.cc:503] Master 127.10.250.126:41709 was elected leader, sending a full tablet report...
I20260620 18:41:44.554131 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.547282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.10.250.65:39303)
0620 18:41:44.550309 (+ 3027us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.550313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.550329 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:41:44.550360 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.550362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.550375 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.550462 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:44.550737 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.551450 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.553141 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.553662 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.553726 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.553767 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.554002 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.554043 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2991,"thread_start_us":96,"threads_started":1}
I20260620 18:41:44.554144 25705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.546988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:39303 (local address 127.0.0.1:42666)
0620 18:41:44.549986 (+ 2998us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.550016 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:44.550075 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:44.550553 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:44.550557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:44.550561 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:44.550639 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.550642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.551550 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.551552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.553032 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.553035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.553749 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.553750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.553849 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.553867 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 18:41:44.553961 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:41:44.554036 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2972,"thread_start_us":93,"threads_started":1}
I20260620 18:41:44.554930 25632 tablet_service.cc:1514] Processing DeleteTablet for tablet 55aaa1917a9d45ae9b0564ed4200fc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:44 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:42666
I20260620 18:41:44.555482 25707 tablet_replica.cc:330] stopping tablet replica
I20260620 18:41:44.555601 25707 raft_consensus.cc:2229] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:44.555693 25707 raft_consensus.cc:2258] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:44.555997 25707 ts_tablet_manager.cc:1905] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:44.557973 25707 ts_tablet_manager.cc:1918] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:44.558056 25707 log.cc:1196] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal/wals/55aaa1917a9d45ae9b0564ed4200fc9b
I20260620 18:41:44.558328 25707 ts_tablet_manager.cc:1939] T 55aaa1917a9d45ae9b0564ed4200fc9b P 5784328fa9b04541acf2c1a5a1cdc9f0: Deleting consensus metadata
I20260620 18:41:44.559047 25496 catalog_manager.cc:4841] TS 5784328fa9b04541acf2c1a5a1cdc9f0 (127.10.250.65:39303): tablet 55aaa1917a9d45ae9b0564ed4200fc9b (table test-table [id=fc3214b2a253453684d4d56efb494a24]) successfully deleted
W20260620 18:41:44.632527 25703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:44.632786 25703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:44.632829 25703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:44.634083 25703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:44.634158 25703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:44.634208 25703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:41:44.636510 25703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:37833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=41295
--tserver_master_addrs=127.10.250.126:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:44.636821 25703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:44.637082 25703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:44.642663 25715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:44.642671 25713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:44.642671 25712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:44.642855 25703 server_base.cc:1034] running on GCE node
I20260620 18:41:44.643168 25703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:44.643419 25703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:44.644599 25703 hybrid_clock.cc:648] HybridClock initialized: now 1781980904644574 us; error 41 us; skew 500 ppm
Jun 20 18:41:44 dist-test-slave-wp6g krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980904, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:44.647285 25703 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:41:44.647876 25703 webserver.cc:460] Webserver started at http://127.10.250.66:41295/ using document root <none> and password file <none>
I20260620 18:41:44.648131 25703 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:44.648228 25703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:44.649739 25703 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 18:41:44.650542 25722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:44.650732 25703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:44.650843 25703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "d00b58e8de1847619ef01f4fefc700e4"
format_stamp: "Formatted at 2026-06-20 18:41:43 on dist-test-slave-wp6g"
server_key: "007269bf68602c27cbb5cc64bd19a2b6"
server_key_iv: "833d1207f833ca04e47e80b1af24c12c"
server_key_version: "encryptionkey@0"
I20260620 18:41:44.651268 25703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:44.668143 25703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:44.670432 25703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:44.670660 25703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:44.670928 25703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:44.671269 25703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:44.671326 25703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:44.671368 25703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:44.671394 25703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:44.677886 25703 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:37833
I20260620 18:41:44.677927 25835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:37833 every 8 connection(s)
I20260620 18:41:44.678342 25703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:41:44 dist-test-slave-wp6g krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980904, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:41:44.688282 25701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.679132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:42081 (local address 127.10.250.126:41709)
0620 18:41:44.679194 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.679196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.679442 (+ 246us) server_negotiation.cc:407] Connection header received
0620 18:41:44.679952 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.679954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.679966 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.680024 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:44.681064 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.681625 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.682778 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.682998 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.683067 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.685294 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:44.685295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:44.685296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:44.685312 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:44.686799 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.687252 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.687254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.687254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.687317 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.687583 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.687584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.687585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.687711 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:44.687789 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.688073 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.688178 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:41:44.688406 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 25703
I20260620 18:41:44.688567 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.679158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:42081)
0620 18:41:44.679305 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.679343 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:44.679851 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:44.680084 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:44.680089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:44.680702 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:44.680910 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.680916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.681733 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.681734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.682675 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.682678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.683083 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.683084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.683191 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.683733 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:44.683742 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:44.685182 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:44.686918 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:44.686926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:44.686929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:44.687171 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:44.687386 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:44.687386 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:44.687387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:44.687513 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:44.687781 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:44.687787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:44.687939 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 18:41:44.688053 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 18:41:44.688252 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260620 18:41:44.688694 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:33425
--local_ip_for_outbound_sockets=127.10.250.67
--tserver_master_addrs=127.10.250.126:41709
--webserver_port=40545
--webserver_interface=127.10.250.67
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:44.689205 25836 heartbeater.cc:347] Connected to a master server at 127.10.250.126:41709
I20260620 18:41:44.689330 25836 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:44.689517 25836 heartbeater.cc:511] Master 127.10.250.126:41709 requested a full tablet report, sending...
I20260620 18:41:44.689960 25510 ts_manager.cc:194] Registered new tserver with Master: d00b58e8de1847619ef01f4fefc700e4 (127.10.250.66:37833)
I20260620 18:41:44.690358 25510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:42081
W20260620 18:41:44.776350 25839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:44.776574 25839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:44.776600 25839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:44.777791 25839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:44.777835 25839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:44.777863 25839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:41:44.779392 25839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:33425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=40545
--tserver_master_addrs=127.10.250.126:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:44.779594 25839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:44.779778 25839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:44.785073 25845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:44.785073 25844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:44.785091 25847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:44.785504 25839 server_base.cc:1034] running on GCE node
I20260620 18:41:44.785724 25839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:44.785996 25839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:44.787206 25839 hybrid_clock.cc:648] HybridClock initialized: now 1781980904787185 us; error 46 us; skew 500 ppm
Jun 20 18:41:44 dist-test-slave-wp6g krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980904, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:44.790122 25839 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:41:44.790704 25839 webserver.cc:460] Webserver started at http://127.10.250.67:40545/ using document root <none> and password file <none>
I20260620 18:41:44.790915 25839 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:44.790970 25839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:44.792522 25839 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:41:44.793231 25854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:44.793398 25839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:44.793491 25839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "1959ea8b02e34e2290cc0e5a0ce27a98"
format_stamp: "Formatted at 2026-06-20 18:41:44 on dist-test-slave-wp6g"
server_key: "49edb73759d44aaba5edc784cdc78096"
server_key_iv: "b550a70a78e4e022902d3eeb1972d2f9"
server_key_version: "encryptionkey@0"
I20260620 18:41:44.793850 25839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:44.808435 25839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:44.810617 25839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:44.810740 25839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:44.810957 25839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:44.811287 25839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:44.811328 25839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:44.811352 25839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:44.811395 25839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:44.817198 25839 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:33425
I20260620 18:41:44.817292 25967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:33425 every 8 connection(s)
I20260620 18:41:44.817646 25839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
Jun 20 18:41:44 dist-test-slave-wp6g krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980904, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:41:44.824412 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 25839
I20260620 18:41:44.828039 25701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.818563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:51769 (local address 127.10.250.126:41709)
0620 18:41:44.818654 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.818656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.818841 (+ 185us) server_negotiation.cc:407] Connection header received
0620 18:41:44.819339 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.819339 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.819350 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.819424 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:44.820500 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.821004 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.822018 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.822210 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.822267 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.824710 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:44.824712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:44.824713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:44.824722 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:44.826303 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.826799 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.826800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.826801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.826899 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.827307 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.827308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.827309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.827430 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:44.827486 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.827872 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.827964 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:41:44.828325 25970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.818462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:51769)
0620 18:41:44.818689 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.818733 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:44.819253 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:44.819557 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:44.819561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:44.820195 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:44.820397 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.820402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.821116 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.821117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.821929 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.821931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.822288 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.822289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.822393 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.822947 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:44.822955 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:44.824500 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:44.826389 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:44.826400 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:44.826403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:44.826724 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:44.826983 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:44.826985 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:44.826985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:44.827225 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:44.827525 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:44.827533 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:44.827720 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 18:41:44.827808 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:41:44.828027 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":136,"threads_started":1}
I20260620 18:41:44.828943 25968 heartbeater.cc:347] Connected to a master server at 127.10.250.126:41709
I20260620 18:41:44.829071 25968 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:44.829289 25968 heartbeater.cc:511] Master 127.10.250.126:41709 requested a full tablet report, sending...
I20260620 18:41:44.829743 25510 ts_manager.cc:194] Registered new tserver with Master: 1959ea8b02e34e2290cc0e5a0ce27a98 (127.10.250.67:33425)
I20260620 18:41:44.830246 25510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:51769
I20260620 18:41:44.832154 25972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.824913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.10.250.126:41709)
0620 18:41:44.825036 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.825037 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.825090 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:41:44.825172 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.825173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.825184 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.825265 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:44.825859 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.826396 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.827594 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.827815 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.827881 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.829437 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:44.829438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:44.829438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:44.829445 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:44.831034 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.831364 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.831365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.831365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.831416 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.831641 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.831642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.831642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.831770 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:44.831896 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.831979 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.832054 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260620 18:41:44.833143 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:41:44.836464 25972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.833706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.10.250.126:41709)
0620 18:41:44.833799 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.833801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.833910 (+ 109us) server_negotiation.cc:407] Connection header received
0620 18:41:44.833997 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.833998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.834048 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.834128 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:44.834294 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.834880 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.835660 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.835865 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.835941 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.836116 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:44.836226 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:44.836337 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.836357 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.836382 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:41:44.836782 25510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:41:44.838615 25510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:41:44.841136 25632 tablet_service.cc:1467] Processing CreateTablet for tablet 500cf253bd04499b960f212ef31eb06d (DEFAULT_TABLE table=test-table [id=222e8768ddd44c42a703c9454b4e0a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:44.841362 25632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500cf253bd04499b960f212ef31eb06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:44.843050 25590 tablet_bootstrap.cc:492] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0: Bootstrap starting.
I20260620 18:41:44.843405 25590 tablet_bootstrap.cc:654] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:44.843935 25590 tablet_bootstrap.cc:492] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0: No bootstrap required, opened a new log
I20260620 18:41:44.844013 25590 ts_tablet_manager.cc:1397] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 18:41:44.844130 25590 raft_consensus.cc:348] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:44.844221 25590 raft_consensus.cc:374] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:44.844280 25590 raft_consensus.cc:729] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5784328fa9b04541acf2c1a5a1cdc9f0, State: Initialized, Role: FOLLOWER
I20260620 18:41:44.844331 25590 consensus_queue.cc:260] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:44.844445 25590 ts_tablet_manager.cc:1428] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:44.845098 25705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.840698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:33425 (local address 127.0.0.1:35484)
0620 18:41:44.840767 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.840780 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:44.840818 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:44.841466 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:44.841467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:44.841469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:44.841508 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.841509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.842551 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.842552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.843649 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.843651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.844802 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.844802 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.844870 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.844875 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:41:44.844936 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 18:41:44.844998 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 18:41:44.845158 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.840845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.10.250.67:33425)
0620 18:41:44.841322 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.841326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.841333 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:41:44.841362 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.841363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.841374 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.841467 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:44.841589 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.842412 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.843752 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.844686 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.844804 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.844869 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.844969 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.845026 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":94,"threads_started":1}
I20260620 18:41:44.845702 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.840698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:37833 (local address 127.0.0.1:57408)
0620 18:41:44.840926 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.840935 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:44.840974 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:44.841357 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:44.841358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:44.841361 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:44.841478 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.841482 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.842877 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.842878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.843938 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.843940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.845431 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.845432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.845526 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.845531 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:41:44.845591 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 18:41:44.845633 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":23,"thread_start_us":88,"threads_started":1}
I20260620 18:41:44.845774 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.840845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.10.250.66:37833)
0620 18:41:44.841069 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.841073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.841083 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:41:44.841118 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.841120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.841136 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.841213 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:44.841578 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.842766 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.844030 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.844955 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.845028 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.845087 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.845640 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.845680 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":117,"threads_started":1}
I20260620 18:41:44.845968 25902 tablet_service.cc:1467] Processing CreateTablet for tablet 500cf253bd04499b960f212ef31eb06d (DEFAULT_TABLE table=test-table [id=222e8768ddd44c42a703c9454b4e0a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:44.846340 25902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500cf253bd04499b960f212ef31eb06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:44.847239 25770 tablet_service.cc:1467] Processing CreateTablet for tablet 500cf253bd04499b960f212ef31eb06d (DEFAULT_TABLE table=test-table [id=222e8768ddd44c42a703c9454b4e0a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:44.847572 25770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500cf253bd04499b960f212ef31eb06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:44.849391 25978 tablet_bootstrap.cc:492] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: Bootstrap starting.
I20260620 18:41:44.850139 25978 tablet_bootstrap.cc:654] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:44.850736 25978 log.cc:824] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:44.850801 25979 tablet_bootstrap.cc:492] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4: Bootstrap starting.
I20260620 18:41:44.851456 25979 tablet_bootstrap.cc:654] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:44.851572 25978 tablet_bootstrap.cc:492] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: No bootstrap required, opened a new log
I20260620 18:41:44.851641 25978 ts_tablet_manager.cc:1397] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:41:44.852034 25979 log.cc:824] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:44.852819 25979 tablet_bootstrap.cc:492] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4: No bootstrap required, opened a new log
I20260620 18:41:44.852919 25979 ts_tablet_manager.cc:1397] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:41:44.853349 25978 raft_consensus.cc:348] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:44.853492 25978 raft_consensus.cc:374] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:44.853515 25978 raft_consensus.cc:729] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1959ea8b02e34e2290cc0e5a0ce27a98, State: Initialized, Role: FOLLOWER
I20260620 18:41:44.853649 25978 consensus_queue.cc:260] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:44.853936 25968 heartbeater.cc:503] Master 127.10.250.126:41709 was elected leader, sending a full tablet report...
I20260620 18:41:44.853940 25978 ts_tablet_manager.cc:1428] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 18:41:44.855070 25979 raft_consensus.cc:348] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:44.855283 25979 raft_consensus.cc:374] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:44.855329 25979 raft_consensus.cc:729] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00b58e8de1847619ef01f4fefc700e4, State: Initialized, Role: FOLLOWER
I20260620 18:41:44.855444 25979 consensus_queue.cc:260] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:44.855751 25979 ts_tablet_manager.cc:1428] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:44.855774 25836 heartbeater.cc:503] Master 127.10.250.126:41709 was elected leader, sending a full tablet report...
W20260620 18:41:44.929015 25837 tablet.cc:2369] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:41:45.030562 25699 tablet.cc:2369] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:45.067618 25982 raft_consensus.cc:482] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:45.067775 25982 raft_consensus.cc:504] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:45.068120 25982 leader_election.cc:288] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d00b58e8de1847619ef01f4fefc700e4 (127.10.250.66:37833), 5784328fa9b04541acf2c1a5a1cdc9f0 (127.10.250.65:39303)
W20260620 18:41:45.068262 25969 tablet.cc:2369] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:45.071583 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.068430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:44743 (local address 127.10.250.66:37833)
0620 18:41:45.068476 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:45.068479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:45.068550 (+ 71us) server_negotiation.cc:407] Connection header received
0620 18:41:45.068648 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:45.068649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:45.068662 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:45.068722 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:45.068956 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.069507 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.070799 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.071247 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.071304 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.071311 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:45.071490 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 18:41:45.071525 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:41:45.071637 25970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.068367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:37833 (local address 127.10.250.67:44743)
0620 18:41:45.068461 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.068476 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:45.068547 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:45.068783 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:45.068784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:45.068786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:45.068869 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:45.068871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.069615 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.069617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:45.070717 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:45.070719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.071318 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.071319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:45.071405 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.071409 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:41:45.071479 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 18:41:45.071520 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:41:45.071980 25984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.068482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:39303 (local address 127.10.250.67:49239)
0620 18:41:45.068644 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.068653 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:45.068743 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:45.068950 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:45.068950 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:45.068952 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:45.069024 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:45.069025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.069892 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.069893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:45.070900 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:45.070902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.071689 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.071690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:45.071783 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.071787 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:41:45.071865 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:41:45.071905 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
I20260620 18:41:45.071996 25985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.068583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:49239 (local address 127.10.250.65:39303)
0620 18:41:45.068710 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:45.068711 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:45.068720 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:41:45.068811 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:45.068812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:45.068823 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:45.068906 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:45.069208 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.069802 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.070998 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.071598 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.071676 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.071686 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:45.071873 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 18:41:45.071924 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260620 18:41:45.072160 25790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500cf253bd04499b960f212ef31eb06d" candidate_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00b58e8de1847619ef01f4fefc700e4" is_pre_election: true
I20260620 18:41:45.072333 25704 raft_consensus.cc:482] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:45.072359 25790 raft_consensus.cc:2454] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1959ea8b02e34e2290cc0e5a0ce27a98 in term 0.
I20260620 18:41:45.072404 25652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500cf253bd04499b960f212ef31eb06d" candidate_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" is_pre_election: true
I20260620 18:41:45.072432 25704 raft_consensus.cc:504] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:45.072563 25652 raft_consensus.cc:2454] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1959ea8b02e34e2290cc0e5a0ce27a98 in term 0.
I20260620 18:41:45.072773 25857 leader_election.cc:302] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1959ea8b02e34e2290cc0e5a0ce27a98, d00b58e8de1847619ef01f4fefc700e4; no voters:
I20260620 18:41:45.072794 25704 leader_election.cc:288] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d00b58e8de1847619ef01f4fefc700e4 (127.10.250.66:37833), 1959ea8b02e34e2290cc0e5a0ce27a98 (127.10.250.67:33425)
I20260620 18:41:45.072906 25982 raft_consensus.cc:2781] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:45.072970 25982 raft_consensus.cc:482] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:45.072999 25982 raft_consensus.cc:3037] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:45.073987 25982 raft_consensus.cc:504] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:45.074133 25982 leader_election.cc:288] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [CANDIDATE]: Term 1 election: Requested vote from peers d00b58e8de1847619ef01f4fefc700e4 (127.10.250.66:37833), 5784328fa9b04541acf2c1a5a1cdc9f0 (127.10.250.65:39303)
I20260620 18:41:45.074282 25652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500cf253bd04499b960f212ef31eb06d" candidate_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0"
I20260620 18:41:45.074326 25790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500cf253bd04499b960f212ef31eb06d" candidate_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00b58e8de1847619ef01f4fefc700e4"
I20260620 18:41:45.074360 25652 raft_consensus.cc:3037] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:45.074405 25790 raft_consensus.cc:3037] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:45.075178 25652 raft_consensus.cc:2454] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1959ea8b02e34e2290cc0e5a0ce27a98 in term 1.
I20260620 18:41:45.075333 25790 raft_consensus.cc:2454] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1959ea8b02e34e2290cc0e5a0ce27a98 in term 1.
I20260620 18:41:45.075515 25857 leader_election.cc:302] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1959ea8b02e34e2290cc0e5a0ce27a98, 5784328fa9b04541acf2c1a5a1cdc9f0; no voters:
I20260620 18:41:45.075662 25982 raft_consensus.cc:2781] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:45.075827 25982 raft_consensus.cc:686] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 1 LEADER]: Becoming Leader. State: Replica: 1959ea8b02e34e2290cc0e5a0ce27a98, State: Running, Role: LEADER
I20260620 18:41:45.075933 25982 consensus_queue.cc:237] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:45.076272 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.073073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:43321 (local address 127.10.250.67:33425)
0620 18:41:45.073170 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:45.073172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:45.073294 (+ 122us) server_negotiation.cc:407] Connection header received
0620 18:41:45.073348 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:45.073349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:45.073358 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:45.073415 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:45.073557 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.073984 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.075359 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.075901 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.075971 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.075981 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:45.076170 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 18:41:45.076214 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:41:45.076359 25987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.072910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:37833 (local address 127.10.250.65:37071)
0620 18:41:45.073094 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.073105 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:45.073154 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:45.073336 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:45.073338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:45.073340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:45.073380 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:45.073382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.074219 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.074220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:45.075315 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:45.075318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.076029 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.076030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:45.076085 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.076091 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:41:45.076197 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:41:45.076233 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":53,"threads_started":1}
I20260620 18:41:45.076540 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.073119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:37071 (local address 127.10.250.66:37833)
0620 18:41:45.073203 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:45.073204 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:45.073211 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:41:45.073233 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:45.073233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:45.073242 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:45.073295 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:45.073448 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.074100 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.075411 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.075937 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.075994 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.076002 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:45.076442 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 18:41:45.076484 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:41:45.076799 25790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500cf253bd04499b960f212ef31eb06d" candidate_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00b58e8de1847619ef01f4fefc700e4" is_pre_election: true
I20260620 18:41:45.076723 25510 catalog_manager.cc:5495] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1959ea8b02e34e2290cc0e5a0ce27a98 (127.10.250.67). New cstate: current_term: 1 leader_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } health_report { overall_health: UNKNOWN } } }
I20260620 18:41:45.076902 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.072921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:33425 (local address 127.10.250.65:43321)
0620 18:41:45.073219 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.073227 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:45.073300 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:45.073419 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:45.073419 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:45.073421 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:45.073482 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:45.073484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.074071 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.074072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:45.075274 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:45.075277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.076016 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.076017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:45.076085 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.076089 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:41:45.076376 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 18:41:45.076489 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":54,"threads_started":1}
I20260620 18:41:45.076936 25790 raft_consensus.cc:2379] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5784328fa9b04541acf2c1a5a1cdc9f0 in current term 1: Already voted for candidate 1959ea8b02e34e2290cc0e5a0ce27a98 in this term.
I20260620 18:41:45.077358 25922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500cf253bd04499b960f212ef31eb06d" candidate_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" is_pre_election: true
I20260620 18:41:45.077656 25585 leader_election.cc:302] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5784328fa9b04541acf2c1a5a1cdc9f0; no voters: 1959ea8b02e34e2290cc0e5a0ce27a98, d00b58e8de1847619ef01f4fefc700e4
I20260620 18:41:45.077807 25704 raft_consensus.cc:2726] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 18:41:45.133442 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.129721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.10.250.67:33425)
0620 18:41:45.129796 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:45.129798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:45.129809 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:41:45.129856 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:45.129859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:45.129884 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:45.130015 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:45.130245 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.131397 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.132232 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.132600 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.132676 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.132917 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:45.133036 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:45.133110 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:45.133263 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 18:41:45.133288 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 18:41:45.135861 25652 raft_consensus.cc:1261] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1959ea8b02e34e2290cc0e5a0ce27a98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:45.136044 25790 raft_consensus.cc:1261] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [term 1 FOLLOWER]: Refusing update from remote peer 1959ea8b02e34e2290cc0e5a0ce27a98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:45.136475 25988 consensus_queue.cc:1035] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:45.136873 25982 consensus_queue.cc:1035] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:45.139743 25991 mvcc.cc:205] Tried to move back new op lower bound from 7298993787432464384 to 7298993787191635968. Current Snapshot: MvccSnapshot[applied={T|T < 7298993787432464384}]
I20260620 18:41:45.140167 25992 mvcc.cc:205] Tried to move back new op lower bound from 7298993787432464384 to 7298993787191635968. Current Snapshot: MvccSnapshot[applied={T|T < 7298993787432464384}]
I20260620 18:41:45.140291 25993 mvcc.cc:205] Tried to move back new op lower bound from 7298993787432464384 to 7298993787191635968. Current Snapshot: MvccSnapshot[applied={T|T < 7298993787432464384}]
I20260620 18:41:45.143263 25509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:45.143388 25509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:45.144229 25509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa: Sending DeleteTablet for 3 replicas of tablet 500cf253bd04499b960f212ef31eb06d
I20260620 18:41:45.144588 25902 tablet_service.cc:1514] Processing DeleteTablet for tablet 500cf253bd04499b960f212ef31eb06d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:45 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:35484
I20260620 18:41:45.144668 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 25548
I20260620 18:41:45.144742 25770 tablet_service.cc:1514] Processing DeleteTablet for tablet 500cf253bd04499b960f212ef31eb06d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:45 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:57408
I20260620 18:41:45.145484 26001 tablet_replica.cc:330] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: stopping tablet replica
I20260620 18:41:45.145612 26001 raft_consensus.cc:2229] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:41:45.145753 26001 raft_consensus.cc:2258] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:45.146337 26001 ts_tablet_manager.cc:1905] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:45.148746 26001 ts_tablet_manager.cc:1918] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:45.148844 26001 log.cc:1196] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/500cf253bd04499b960f212ef31eb06d
I20260620 18:41:45.149130 26001 ts_tablet_manager.cc:1939] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: Deleting consensus metadata
I20260620 18:41:45.149884 25495 catalog_manager.cc:4841] TS 1959ea8b02e34e2290cc0e5a0ce27a98 (127.10.250.67:33425): tablet 500cf253bd04499b960f212ef31eb06d (table test-table [id=222e8768ddd44c42a703c9454b4e0a00]) successfully deleted
I20260620 18:41:45.150686 26002 tablet_replica.cc:330] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4: stopping tablet replica
I20260620 18:41:45.150835 26002 raft_consensus.cc:2229] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:45.150959 26002 raft_consensus.cc:2258] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 18:41:45.151013 25496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:39303 (error 108)
I20260620 18:41:45.151494 26002 ts_tablet_manager.cc:1905] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 18:41:45.151667 25974 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:45.151404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:39303 (local address 127.0.0.1:42680)
0620 18:41:45.151477 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.151490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:45.151554 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:45.151608 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:39303: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 18:41:45.151922 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 25703
W20260620 18:41:45.151952 25496 catalog_manager.cc:4584] TS 5784328fa9b04541acf2c1a5a1cdc9f0 (127.10.250.65:39303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 500cf253bd04499b960f212ef31eb06d: Network error: Client connection negotiation failed: client connection to 127.10.250.65:39303: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260620 18:41:45.158080 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 25839
I20260620 18:41:45.158188 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.158035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:37833 (local address 127.0.0.1:57422)
0620 18:41:45.158086 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.158140 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.66:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 18:41:45.162890 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 25477
I20260620 18:41:45.168493 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:41709
--webserver_interface=127.10.250.126
--webserver_port=42583
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:41709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:45.254485 26004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:45.254815 26004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:45.254840 26004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:45.256160 26004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:45.256213 26004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:45.256228 26004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:45.256242 26004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:45.256254 26004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:45.257872 26004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:41709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:41709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=42583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:45.258154 26004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:45.258440 26004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:45.264256 26010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:45.264257 26012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:45.264256 26009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:45.264276 26004 server_base.cc:1034] running on GCE node
I20260620 18:41:45.264802 26004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:45.265144 26004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:45.266299 26004 hybrid_clock.cc:648] HybridClock initialized: now 1781980905266272 us; error 48 us; skew 500 ppm
Jun 20 18:41:45 dist-test-slave-wp6g krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980905, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:45.269404 26004 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:41:45.270002 26004 webserver.cc:460] Webserver started at http://127.10.250.126:42583/ using document root <none> and password file <none>
I20260620 18:41:45.270251 26004 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:45.270327 26004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:45.271915 26004 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:45.272943 26019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:45.273222 26004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:45.273340 26004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "ba8e17c78f2e4468a94d06ab18a312aa"
format_stamp: "Formatted at 2026-06-20 18:41:43 on dist-test-slave-wp6g"
server_key: "af67454624bd8c6e84a9fb15ad381d43"
server_key_iv: "220e416b7ba12fa587ec5d09256a0dde"
server_key_version: "encryptionkey@0"
I20260620 18:41:45.273761 26004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:45.293531 26004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:45.296345 26004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:45.296547 26004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:45.301064 26004 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:41709
I20260620 18:41:45.301100 26071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:41709 every 8 connection(s)
I20260620 18:41:45.301512 26004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:45.303251 26072 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:41:45.304035 26072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa: Bootstrap starting.
I20260620 18:41:45.304570 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 26004
I20260620 18:41:45.304855 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:39303
--local_ip_for_outbound_sockets=127.10.250.65
--tserver_master_addrs=127.10.250.126:41709
--webserver_port=38285
--webserver_interface=127.10.250.65
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:45.313540 26072 log.cc:824] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:45.316028 26072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:41:45.316377 26072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa: Bootstrap complete.
I20260620 18:41:45.318815 26072 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } }
I20260620 18:41:45.319051 26072 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8e17c78f2e4468a94d06ab18a312aa, State: Initialized, Role: FOLLOWER
I20260620 18:41:45.319221 26072 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } }
I20260620 18:41:45.319294 26072 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:45.319355 26072 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:45.319403 26072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:41:45.320458 26072 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } }
I20260620 18:41:45.320551 26072 leader_election.cc:302] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba8e17c78f2e4468a94d06ab18a312aa; no voters:
I20260620 18:41:45.320786 26072 leader_election.cc:288] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 18:41:45.320827 26076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:41:45.320935 26076 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [term 3 LEADER]: Becoming Leader. State: Replica: ba8e17c78f2e4468a94d06ab18a312aa, State: Running, Role: LEADER
I20260620 18:41:45.321084 26072 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:45.321126 26076 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } }
I20260620 18:41:45.321484 26078 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba8e17c78f2e4468a94d06ab18a312aa. Latest consensus state: current_term: 3 leader_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } } }
I20260620 18:41:45.321555 26078 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:45.321736 26077 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8e17c78f2e4468a94d06ab18a312aa" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 41709 } } }
I20260620 18:41:45.321830 26077 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8e17c78f2e4468a94d06ab18a312aa [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:45.322083 26080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:45.322968 26080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:45.323143 26080 catalog_manager.cc:1259] Loaded cluster ID: d6d67effd18a4cdcbb4be72224cdb4ae
I20260620 18:41:45.323375 26080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 18:41:45.323606 26080 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0380b8e9ad google::LogMessage::Fail() at ??:0
@ 0x7f0380b9188a google::LogMessage::SendToLog() at ??:0
@ 0x7f0380b8e4de google::LogMessage::Flush() at ??:0
@ 0x7f0380b8fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0388aaf0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f0388ab88b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0382a5c0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0382a51954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f03838b56db start_thread at ??:0
@ 0x7f037f68c61f clone at ??:0
W20260620 18:41:45.404891 26074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:45.405107 26074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:45.405131 26074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:45.406334 26074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:45.406381 26074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:45.406409 26074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:41:45.407953 26074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:39303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=38285
--tserver_master_addrs=127.10.250.126:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:45.408229 26074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:45.408488 26074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:45.413879 26111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:45.413879 26108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:45.413866 26109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:45.414489 26074 server_base.cc:1034] running on GCE node
I20260620 18:41:45.414767 26074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:45.415037 26074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:45.416224 26074 hybrid_clock.cc:648] HybridClock initialized: now 1781980905416208 us; error 32 us; skew 500 ppm
Jun 20 18:41:45 dist-test-slave-wp6g krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980905, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:45.418933 26074 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:41:45.419405 26074 webserver.cc:460] Webserver started at http://127.10.250.65:38285/ using document root <none> and password file <none>
I20260620 18:41:45.419646 26074 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:45.419721 26074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:45.421233 26074 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 18:41:45.422003 26118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:45.422219 26074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:45.422324 26074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "5784328fa9b04541acf2c1a5a1cdc9f0"
format_stamp: "Formatted at 2026-06-20 18:41:43 on dist-test-slave-wp6g"
server_key: "b82508a5b5e3fcb194b630637b71d766"
server_key_iv: "6a6cc96ede1d7def88eef8d2f719c228"
server_key_version: "encryptionkey@0"
I20260620 18:41:45.422786 26074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:45.447350 26074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:45.449877 26074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:45.450052 26074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:45.450318 26074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:45.450908 26125 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 18:41:45.452667 26074 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 18:41:45.452756 26074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 18:41:45.452809 26074 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 18:41:45.453529 26074 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 18:41:45.453604 26074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 18:41:45.453692 26125 tablet_bootstrap.cc:492] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0: Bootstrap starting.
I20260620 18:41:45.461026 26074 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:39303
I20260620 18:41:45.461177 26232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:39303 every 8 connection(s)
I20260620 18:41:45.461519 26074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:41:45.465755 26125 log.cc:824] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:45.469022 26125 tablet_bootstrap.cc:492] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:41:45.469430 26125 tablet_bootstrap.cc:492] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0: Bootstrap complete.
I20260620 18:41:45.470728 26125 ts_tablet_manager.cc:1397] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.011s
I20260620 18:41:45.471000 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 26074
I20260620 18:41:45.471252 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:37833
--local_ip_for_outbound_sockets=127.10.250.66
--tserver_master_addrs=127.10.250.126:41709
--webserver_port=41295
--webserver_interface=127.10.250.66
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:45.472359 26125 raft_consensus.cc:348] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:45.472776 26125 raft_consensus.cc:729] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5784328fa9b04541acf2c1a5a1cdc9f0, State: Initialized, Role: FOLLOWER
I20260620 18:41:45.472962 26125 consensus_queue.cc:260] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:45.473320 26125 ts_tablet_manager.cc:1428] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
W20260620 18:41:45.569216 26237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:45.569505 26237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:45.569784 26237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:45.571548 26237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:45.571645 26237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:45.571718 26237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:41:45.573555 26237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:37833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=41295
--tserver_master_addrs=127.10.250.126:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:45.573920 26237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:45.574204 26237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:45.580691 26243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:45.580937 26237 server_base.cc:1034] running on GCE node
W20260620 18:41:45.580684 26244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:45.580687 26246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:45.581476 26237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:45.581826 26237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260620 18:41:45.582175 26235 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:45.462284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:40819)
0620 18:41:45.462621 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.462672 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:45.463353 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:45.581892 (+118539us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":118,"threads_started":1}
W20260620 18:41:45.582433 26233 heartbeater.cc:644] Failed to heartbeat to 127.10.250.126:41709 (0 consecutive failures): Network error: Failed to ping master at 127.10.250.126:41709: Client connection negotiation failed: client connection to 127.10.250.126:41709: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 18:41:45.582823 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.582698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:40475)
0620 18:41:45.582745 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.582788 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 18:41:45.583040 26237 hybrid_clock.cc:648] HybridClock initialized: now 1781980905583017 us; error 36 us; skew 500 ppm
I20260620 18:41:45.583335 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.583235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:47119)
0620 18:41:45.583277 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.583309 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 18:41:45.583468 26233 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 18:41:45 dist-test-slave-wp6g krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980905, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:45.585721 26237 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:41:45.586164 26237 webserver.cc:460] Webserver started at http://127.10.250.66:41295/ using document root <none> and password file <none>
I20260620 18:41:45.586376 26237 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:45.586421 26237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:45.587899 26237 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:45.588714 26253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:45.588927 26237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:45.589033 26237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "d00b58e8de1847619ef01f4fefc700e4"
format_stamp: "Formatted at 2026-06-20 18:41:43 on dist-test-slave-wp6g"
server_key: "007269bf68602c27cbb5cc64bd19a2b6"
server_key_iv: "833d1207f833ca04e47e80b1af24c12c"
server_key_version: "encryptionkey@0"
I20260620 18:41:45.589610 26237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:45.612702 26237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:45.615465 26237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:45.615645 26237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:45.615896 26237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:45.616348 26260 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 18:41:45.617918 26237 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 18:41:45.617985 26237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 18:41:45.618047 26237 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 18:41:45.618911 26237 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 18:41:45.618985 26237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:45.619105 26260 tablet_bootstrap.cc:492] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4: Bootstrap starting.
I20260620 18:41:45.620651 26260 log.cc:824] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:45.624183 26260 tablet_bootstrap.cc:492] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:41:45.624542 26260 tablet_bootstrap.cc:492] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4: Bootstrap complete.
I20260620 18:41:45.624720 26260 ts_tablet_manager.cc:1397] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 18:41:45.626255 26237 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:37833
I20260620 18:41:45.626327 26368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:37833 every 8 connection(s)
I20260620 18:41:45.626689 26237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:41:45.627025 26260 raft_consensus.cc:348] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:45.627348 26260 raft_consensus.cc:729] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00b58e8de1847619ef01f4fefc700e4, State: Initialized, Role: FOLLOWER
I20260620 18:41:45.627514 26260 consensus_queue.cc:260] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:45.627789 26260 ts_tablet_manager.cc:1428] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20260620 18:41:45.628075 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 26237
I20260620 18:41:45.628170 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.627526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:34925)
0620 18:41:45.627949 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.628017 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":110,"threads_started":1}
I20260620 18:41:45.628361 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:33425
--local_ip_for_outbound_sockets=127.10.250.67
--tserver_master_addrs=127.10.250.126:41709
--webserver_port=40545
--webserver_interface=127.10.250.67
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:45.628391 26369 heartbeater.cc:644] Failed to heartbeat to 127.10.250.126:41709 (0 consecutive failures): Network error: Failed to ping master at 127.10.250.126:41709: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
I20260620 18:41:45.628891 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.628684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:41935)
0620 18:41:45.628821 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.628859 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260620 18:41:45.629237 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.629123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:55637)
0620 18:41:45.629189 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.629213 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:41:45.629350 26369 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:41:45.713377 26373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:45.713653 26373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:45.713699 26373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:45.715071 26373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:45.715162 26373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:45.715212 26373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:41:45.716851 26373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:33425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=40545
--tserver_master_addrs=127.10.250.126:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:41:45.717167 26373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:45.717427 26373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:45.722932 26381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:45.723016 26373 server_base.cc:1034] running on GCE node
W20260620 18:41:45.722930 26379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:45.723126 26378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:45.723343 26373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:45.723662 26373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:45.724839 26373 hybrid_clock.cc:648] HybridClock initialized: now 1781980905724833 us; error 38 us; skew 500 ppm
Jun 20 18:41:45 dist-test-slave-wp6g krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980905, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:45.727751 26373 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:41:45.728281 26373 webserver.cc:460] Webserver started at http://127.10.250.67:40545/ using document root <none> and password file <none>
I20260620 18:41:45.728516 26373 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:45.728590 26373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:45.730074 26373 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:41:45.730787 26388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:45.730969 26373 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:41:45.731074 26373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "1959ea8b02e34e2290cc0e5a0ce27a98"
format_stamp: "Formatted at 2026-06-20 18:41:44 on dist-test-slave-wp6g"
server_key: "49edb73759d44aaba5edc784cdc78096"
server_key_iv: "b550a70a78e4e022902d3eeb1972d2f9"
server_key_version: "encryptionkey@0"
I20260620 18:41:45.731438 26373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:45.745219 26373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:45.747720 26373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:45.747895 26373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:45.748147 26373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:45.748476 26373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:45.748530 26373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:45.748571 26373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:45.748600 26373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:45.754949 26373 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:33425
I20260620 18:41:45.755052 26501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:33425 every 8 connection(s)
I20260620 18:41:45.755466 26373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:41:45.756724 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.756265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:50975)
0620 18:41:45.756458 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.756548 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
W20260620 18:41:45.757267 26502 heartbeater.cc:644] Failed to heartbeat to 127.10.250.126:41709 (0 consecutive failures): Network error: Failed to ping master at 127.10.250.126:41709: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
I20260620 18:41:45.757606 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.757502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:53875)
0620 18:41:45.757551 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.757584 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 18:41:45.757941 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.757863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:60865)
0620 18:41:45.757903 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.757926 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 18:41:45.758081 26502 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:41:45.764312 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 26373
I20260620 18:41:46.584215 26506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.583909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:35903)
0620 18:41:46.584053 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.584097 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":100,"threads_started":1}
I20260620 18:41:46.630060 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.629748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:45693)
0620 18:41:46.629915 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.629960 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":76,"threads_started":1}
I20260620 18:41:46.758858 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.758466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:56029)
0620 18:41:46.758755 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.758797 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":215,"threads_started":1}
I20260620 18:41:46.896247 26509 raft_consensus.cc:482] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:46.896368 26509 raft_consensus.cc:504] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:46.896811 26509 leader_election.cc:288] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d00b58e8de1847619ef01f4fefc700e4 (127.10.250.66:37833), 1959ea8b02e34e2290cc0e5a0ce27a98 (127.10.250.67:33425)
Jun 20 18:41:46 dist-test-slave-wp6g krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980905, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.66@KRBTEST.COM
Jun 20 18:41:46 dist-test-slave-wp6g krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980905, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.67@KRBTEST.COM
I20260620 18:41:46.908448 26506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.897038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:37833 (local address 127.10.250.65:52439)
0620 18:41:46.897095 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.897108 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:46.897172 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:46.897696 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:46.897698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:46.898015 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:46.898201 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.898207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.899368 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.899369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.900166 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.900168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.900750 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.900751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.900875 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.901536 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:46.901540 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:46.903309 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:46.906199 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:46.906210 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:46.906212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:46.906734 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:46.907034 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:46.907035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:46.907035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:46.907220 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:46.907770 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:46.907779 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:46.907960 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 18:41:46.908132 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 18:41:46.908320 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":29}
I20260620 18:41:46.908481 26512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.897197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:37101 (local address 127.10.250.67:33425)
0620 18:41:46.897366 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.897377 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.897415 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:41:46.897848 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.897850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.897884 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.898042 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:46.898390 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.899319 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.900272 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.900696 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.900817 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.903686 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:46.903690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:46.903691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:46.903704 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:46.906032 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:46.906826 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:46.906830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:46.906830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:46.906953 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:46.907301 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:46.907302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:46.907302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:46.907660 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:46.907755 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.908236 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.908365 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":65,"threads_started":1}
I20260620 18:41:46.908481 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.897167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:52439 (local address 127.10.250.66:37833)
0620 18:41:46.897348 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.897359 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.897378 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:41:46.897482 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.897484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.897523 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.897670 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:46.898315 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.899253 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.900258 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.900658 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.900786 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.903452 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:46.903455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:46.903455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:46.903466 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:46.906048 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:46.906833 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:46.906836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:46.906836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:46.906952 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:46.907301 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:46.907302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:46.907302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:46.907697 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:46.907772 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.908224 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.908366 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
I20260620 18:41:46.908521 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.897056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:33425 (local address 127.10.250.65:37101)
0620 18:41:46.897255 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.897265 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:46.897324 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:46.898020 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:46.898021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:46.898223 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:46.898268 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.898270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.899417 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.899418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.900165 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.900167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.900785 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.900786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.900876 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.901489 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:46.901498 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:46.903586 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:46.906199 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:46.906210 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:46.906212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:46.906734 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:46.907034 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:46.907035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:46.907035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:46.907220 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:46.907770 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:46.907786 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:46.907960 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 18:41:46.908097 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 18:41:46.908379 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":239,"thread_start_us":50,"threads_started":1}
I20260620 18:41:46.909956 26323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500cf253bd04499b960f212ef31eb06d" candidate_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d00b58e8de1847619ef01f4fefc700e4" is_pre_election: true
I20260620 18:41:46.910138 26323 raft_consensus.cc:2454] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5784328fa9b04541acf2c1a5a1cdc9f0 in term 1.
W20260620 18:41:46.910337 26120 leader_election.cc:341] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1959ea8b02e34e2290cc0e5a0ce27a98 (127.10.250.67:33425): Not found: Tablet not found: 500cf253bd04499b960f212ef31eb06d
I20260620 18:41:46.910451 26121 leader_election.cc:302] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5784328fa9b04541acf2c1a5a1cdc9f0, d00b58e8de1847619ef01f4fefc700e4; no voters: 1959ea8b02e34e2290cc0e5a0ce27a98
I20260620 18:41:46.910665 26509 raft_consensus.cc:2781] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 18:41:46.910729 26509 raft_consensus.cc:482] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:46.910746 26509 raft_consensus.cc:3037] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:41:46.911960 26509 raft_consensus.cc:504] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:46.912129 26509 leader_election.cc:288] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [CANDIDATE]: Term 2 election: Requested vote from peers d00b58e8de1847619ef01f4fefc700e4 (127.10.250.66:37833), 1959ea8b02e34e2290cc0e5a0ce27a98 (127.10.250.67:33425)
I20260620 18:41:46.912346 26323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500cf253bd04499b960f212ef31eb06d" candidate_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d00b58e8de1847619ef01f4fefc700e4"
I20260620 18:41:46.912464 26323 raft_consensus.cc:3037] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [term 1 FOLLOWER]: Advancing to term 2
W20260620 18:41:46.912493 26120 leader_election.cc:341] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 1959ea8b02e34e2290cc0e5a0ce27a98 (127.10.250.67:33425): Not found: Tablet not found: 500cf253bd04499b960f212ef31eb06d
I20260620 18:41:46.913981 26323 raft_consensus.cc:2454] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5784328fa9b04541acf2c1a5a1cdc9f0 in term 2.
I20260620 18:41:46.914213 26121 leader_election.cc:302] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5784328fa9b04541acf2c1a5a1cdc9f0, d00b58e8de1847619ef01f4fefc700e4; no voters: 1959ea8b02e34e2290cc0e5a0ce27a98
I20260620 18:41:46.914354 26509 raft_consensus.cc:2781] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:41:46.914463 26509 raft_consensus.cc:686] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [term 2 LEADER]: Becoming Leader. State: Replica: 5784328fa9b04541acf2c1a5a1cdc9f0, State: Running, Role: LEADER
I20260620 18:41:46.914614 26509 consensus_queue.cc:237] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:46.914932 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.914724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:49985)
0620 18:41:46.914851 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.914890 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 18:41:47.443347 26323 raft_consensus.cc:1261] T 500cf253bd04499b960f212ef31eb06d P d00b58e8de1847619ef01f4fefc700e4 [term 2 FOLLOWER]: Refusing update from remote peer 5784328fa9b04541acf2c1a5a1cdc9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 18:41:47.443779 26515 consensus_queue.cc:1035] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 18:41:47.444968 26518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:47.444767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:60939)
0620 18:41:47.444870 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:47.444919 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":49,"threads_started":1}
I20260620 18:41:47.445971 26518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:47.445868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:38497)
0620 18:41:47.445904 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:47.445938 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260620 18:41:47.446014 26120 consensus_peers.cc:487] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0 -> Peer 1959ea8b02e34e2290cc0e5a0ce27a98 (127.10.250.67:33425): Couldn't send request to peer 1959ea8b02e34e2290cc0e5a0ce27a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 500cf253bd04499b960f212ef31eb06d. This is attempt 1: this message will repeat every 5th retry.
I20260620 18:41:47.446019 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:47.445781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:40183)
0620 18:41:47.445949 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:47.445978 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
I20260620 18:41:47.759666 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:47.759389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:53373)
0620 18:41:47.759557 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:47.759618 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260620 18:41:48.067466 26526 ts_tablet_manager.cc:927] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: Initiating tablet copy from peer 5784328fa9b04541acf2c1a5a1cdc9f0 (127.10.250.65:39303)
I20260620 18:41:48.067764 26526 tablet_copy_client.cc:316] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: tablet copy: Beginning tablet copy session from remote peer at address 127.10.250.65:39303
Jun 20 18:41:48 dist-test-slave-wp6g krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980905, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.65@KRBTEST.COM
I20260620 18:41:48.076608 26528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:48.068244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:58497 (local address 127.10.250.65:39303)
0620 18:41:48.068431 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:48.068435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:48.068445 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:41:48.068468 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:48.068470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:48.068485 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:48.068574 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:48.069086 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:48.069777 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:48.070635 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:48.070955 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:48.071059 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:48.073162 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:48.073172 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:48.073173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:48.073181 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:48.075123 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:48.075650 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:48.075655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:48.075655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:48.075728 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:48.075944 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:48.075945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:48.075946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:48.076119 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:48.076219 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:48.076405 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 18:41:48.076503 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":52,"threads_started":1}
I20260620 18:41:48.076624 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:48.068107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:39303 (local address 127.10.250.67:58497)
0620 18:41:48.068167 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:48.068185 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:48.068251 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:48.068578 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:48.068587 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:48.068885 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:48.068976 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:48.068979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:48.069864 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:48.069865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:48.070464 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:48.070466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:48.071060 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:48.071061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:48.071149 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:48.071637 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:48.071642 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:48.073053 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:48.075229 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:48.075240 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:48.075243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:48.075549 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:48.075811 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:48.075812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:48.075812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:48.075875 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:48.076195 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:48.076203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:48.076265 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 18:41:48.076367 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:41:48.076501 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 18:41:48.077275 26207 tablet_copy_service.cc:140] P 5784328fa9b04541acf2c1a5a1cdc9f0: Received BeginTabletCopySession request for tablet 500cf253bd04499b960f212ef31eb06d from peer 1959ea8b02e34e2290cc0e5a0ce27a98 ({username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:58497)
I20260620 18:41:48.077365 26207 tablet_copy_service.cc:161] P 5784328fa9b04541acf2c1a5a1cdc9f0: Beginning new tablet copy session on tablet 500cf253bd04499b960f212ef31eb06d from peer 1959ea8b02e34e2290cc0e5a0ce27a98 at {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:58497: session id = 1959ea8b02e34e2290cc0e5a0ce27a98-500cf253bd04499b960f212ef31eb06d
I20260620 18:41:48.077919 26207 tablet_copy_source_session.cc:205] T 500cf253bd04499b960f212ef31eb06d P 5784328fa9b04541acf2c1a5a1cdc9f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:41:48.079175 26526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500cf253bd04499b960f212ef31eb06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:48.081924 26526 tablet_copy_client.cc:791] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: tablet copy: Starting download of 0 data blocks...
I20260620 18:41:48.082108 26526 tablet_copy_client.cc:655] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: tablet copy: Starting download of 1 WAL segments...
I20260620 18:41:48.083351 26526 tablet_copy_client.cc:531] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:41:48.084192 26526 tablet_bootstrap.cc:492] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: Bootstrap starting.
I20260620 18:41:48.085747 26526 log.cc:824] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:48.088491 26526 tablet_bootstrap.cc:492] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:41:48.088738 26526 tablet_bootstrap.cc:492] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: Bootstrap complete.
I20260620 18:41:48.088888 26526 ts_tablet_manager.cc:1397] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 18:41:48.089883 26526 raft_consensus.cc:348] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:48.090083 26526 raft_consensus.cc:729] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1959ea8b02e34e2290cc0e5a0ce27a98, State: Initialized, Role: FOLLOWER
I20260620 18:41:48.090258 26526 consensus_queue.cc:260] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00b58e8de1847619ef01f4fefc700e4" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37833 } } peers { permanent_uuid: "1959ea8b02e34e2290cc0e5a0ce27a98" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33425 } } peers { permanent_uuid: "5784328fa9b04541acf2c1a5a1cdc9f0" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 39303 } }
I20260620 18:41:48.090631 26526 ts_tablet_manager.cc:1428] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260620 18:41:48.090992 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:48.090855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:60611)
0620 18:41:48.090928 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:48.090967 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 18:41:48.090996 26207 tablet_copy_service.cc:342] P 5784328fa9b04541acf2c1a5a1cdc9f0: Request end of tablet copy session 1959ea8b02e34e2290cc0e5a0ce27a98-500cf253bd04499b960f212ef31eb06d received from {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:58497
I20260620 18:41:48.091104 26207 tablet_copy_service.cc:433] P 5784328fa9b04541acf2c1a5a1cdc9f0: ending tablet copy session 1959ea8b02e34e2290cc0e5a0ce27a98-500cf253bd04499b960f212ef31eb06d on tablet 500cf253bd04499b960f212ef31eb06d with peer 1959ea8b02e34e2290cc0e5a0ce27a98
I20260620 18:41:48.091363 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:48.091258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:54889)
0620 18:41:48.091313 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:48.091339 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 18:41:48.446811 26531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:48.446494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:52765)
0620 18:41:48.446685 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:48.446754 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":143,"threads_started":1}
I20260620 18:41:48.446817 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:48.446624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:55841)
0620 18:41:48.446743 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:48.446777 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
I20260620 18:41:48.538110 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:48.537920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:47049)
0620 18:41:48.538025 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:48.538064 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 18:41:48.538158 26456 raft_consensus.cc:1203] T 500cf253bd04499b960f212ef31eb06d P 1959ea8b02e34e2290cc0e5a0ce27a98 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 18:41:48.538707 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:48.538599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:44953)
0620 18:41:48.538648 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:48.538679 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 18:41:49.447656 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:49.447423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:34761)
0620 18:41:49.447568 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:49.447611 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260620 18:41:49.538863 26535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:49.538598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:49193)
0620 18:41:49.538767 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:49.538817 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260620 18:41:49.539423 26536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:49.539229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:37515)
0620 18:41:49.539349 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:49.539386 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260620 18:41:50.448544 26537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.448272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:51835)
0620 18:41:50.448448 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.448495 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
I20260620 18:41:50.539682 26538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.539450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:59741)
0620 18:41:50.539594 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.539635 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":60,"threads_started":1}
I20260620 18:41:50.540148 26538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.540045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:33355)
0620 18:41:50.540094 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.540125 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 18:41:50.540156 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.539895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:41855)
0620 18:41:50.540069 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.540108 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":54,"threads_started":1}
I20260620 18:41:51.449510 26541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:51.449212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:51237)
0620 18:41:51.449356 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:51.449396 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260620 18:41:51.540972 26542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:51.540726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:38187)
0620 18:41:51.540899 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:51.540932 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":81,"threads_started":1}
I20260620 18:41:51.540969 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:51.540726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:57849)
0620 18:41:51.540871 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:51.540922 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260620 18:41:52.450379 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:52.450126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:56261)
0620 18:41:52.450285 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:52.450330 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260620 18:41:52.541790 26546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:52.541553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:32843)
0620 18:41:52.541699 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:52.541744 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260620 18:41:52.541791 26547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:52.541553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:37041)
0620 18:41:52.541717 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:52.541750 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260620 18:41:53.451174 26548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:53.450968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:51799)
0620 18:41:53.451098 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:53.451127 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260620 18:41:53.542657 26550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:53.542406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:54433)
0620 18:41:53.542567 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:53.542610 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
I20260620 18:41:53.542665 26549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:53.542379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:45085)
0620 18:41:53.542560 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:53.542609 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260620 18:41:54.452047 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:54.451765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:33313)
0620 18:41:54.451948 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:54.451998 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260620 18:41:54.543442 26552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:54.543147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:38257)
0620 18:41:54.543303 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:54.543358 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
I20260620 18:41:54.543442 26553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:54.543202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:48779)
0620 18:41:54.543360 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:54.543397 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
I20260620 18:41:55.452924 26554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:55.452634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:52831)
0620 18:41:55.452822 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:55.452872 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260620 18:41:55.544243 26556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:55.544000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:47643)
0620 18:41:55.544147 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:55.544197 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260620 18:41:55.544261 26555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:55.544000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:44271)
0620 18:41:55.544171 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:55.544220 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":77,"threads_started":1}
I20260620 18:41:56.453814 26557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:56.453531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:34165)
0620 18:41:56.453714 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:56.453766 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":71,"threads_started":1}
I20260620 18:41:56.545048 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:56.544785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:56003)
0620 18:41:56.544952 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:56.544995 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
I20260620 18:41:56.545085 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:56.544800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:38465)
0620 18:41:56.545000 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:56.545042 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":87,"threads_started":1}
I20260620 18:41:57.454809 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:57.454494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:41055)
0620 18:41:57.454700 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:57.454755 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":130,"threads_started":1}
I20260620 18:41:57.545857 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:57.545620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:52813)
0620 18:41:57.545765 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:57.545802 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260620 18:41:57.545990 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:57.545620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:55451)
0620 18:41:57.545881 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:57.545936 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":205,"threads_started":1}
I20260620 18:41:58.455940 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:58.455588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:57545)
0620 18:41:58.455841 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:58.455888 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":70,"threads_started":1}
I20260620 18:41:58.546756 26564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:58.546489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:32937)
0620 18:41:58.546679 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:58.546713 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":100,"threads_started":1}
I20260620 18:41:58.546739 26565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:58.546505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:45921)
0620 18:41:58.546648 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:58.546692 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260620 18:41:59.456918 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:59.456619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:33151)
0620 18:41:59.456819 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:59.456868 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260620 18:41:59.547453 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:59.547227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:42225)
0620 18:41:59.547363 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:59.547407 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260620 18:41:59.547497 26569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:59.547279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:48225)
0620 18:41:59.547422 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:59.547456 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260620 18:42:00.457770 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:00.457530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.66:43631)
0620 18:42:00.457675 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:00.457722 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260620 18:42:00.548293 26572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:00.548058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.65:46531)
0620 18:42:00.548200 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:00.548247 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260620 18:42:00.548326 26571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:00.548058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:41709 (local address 127.10.250.67:44657)
0620 18:42:00.548252 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:00.548286 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260620 18:42:00.768687 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 26074
I20260620 18:42:00.773881 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 26237
I20260620 18:42:00.778781 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 26373
2026-06-20T18:42:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18531 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:42:00 dist-test-slave-wp6g krb5kdc[26575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:42:00 dist-test-slave-wp6g krb5kdc[26575](info): set up 2 sockets
Jun 20 18:42:00 dist-test-slave-wp6g krb5kdc[26575](info): commencing operation
krb5kdc: starting...
W20260620 18:42:01.854988 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:42:01 dist-test-slave-wp6g krb5kdc[26575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:42:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:42:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:42:01.987339 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44805
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:44805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:02.071466 26588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:02.071707 26588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:02.071731 26588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:02.073005 26588 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:42:02.073053 26588 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:42:02.073066 26588 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:42:02.073078 26588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:02.073092 26588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:02.073105 26588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:02.073163 26588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:02.073180 26588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:02.074832 26588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:44805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:42:02.075124 26588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:02.075402 26588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:02.081566 26594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:02.081687 26593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:02.081717 26596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:02.081579 26588 server_base.cc:1034] running on GCE node
I20260620 18:42:02.082285 26588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:02.082705 26588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:02.083942 26588 hybrid_clock.cc:648] HybridClock initialized: now 1781980922083915 us; error 41 us; skew 500 ppm
Jun 20 18:42:02 dist-test-slave-wp6g krb5kdc[26575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980922, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:02.087280 26588 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:42:02.087908 26588 webserver.cc:460] Webserver started at http://127.10.250.126:44125/ using document root <none> and password file <none>
I20260620 18:42:02.088171 26588 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:02.088222 26588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:02.088388 26588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:02.089530 26588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "cae28bf3223a4903a5ee281684216b1f"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "88be9f89db06ad092804d321d8782e68"
server_key_iv: "1ae82d7d2a8c3007e1e9ca2f8ab02d24"
server_key_version: "encryptionkey@0"
I20260620 18:42:02.090142 26588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "cae28bf3223a4903a5ee281684216b1f"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "88be9f89db06ad092804d321d8782e68"
server_key_iv: "1ae82d7d2a8c3007e1e9ca2f8ab02d24"
server_key_version: "encryptionkey@0"
I20260620 18:42:02.092060 26588 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 18:42:02.093062 26603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:02.093344 26588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:42:02.093456 26588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "cae28bf3223a4903a5ee281684216b1f"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "88be9f89db06ad092804d321d8782e68"
server_key_iv: "1ae82d7d2a8c3007e1e9ca2f8ab02d24"
server_key_version: "encryptionkey@0"
I20260620 18:42:02.093544 26588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:02.107368 26588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:02.110059 26588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:02.110282 26588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:02.115082 26588 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:44805
I20260620 18:42:02.115254 26655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:44805 every 8 connection(s)
I20260620 18:42:02.115586 26588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:02.116565 26656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:02.119182 26656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Bootstrap starting.
I20260620 18:42:02.119820 26656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:02.120132 26656 log.cc:824] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:02.120998 26656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: No bootstrap required, opened a new log
I20260620 18:42:02.122947 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 26588
I20260620 18:42:02.123041 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:42:02.122915 26656 raft_consensus.cc:348] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } }
I20260620 18:42:02.123118 26656 raft_consensus.cc:374] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:02.123170 26656 raft_consensus.cc:729] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae28bf3223a4903a5ee281684216b1f, State: Initialized, Role: FOLLOWER
I20260620 18:42:02.123239 11241 external_mini_cluster.cc:1428] Setting key a294b5a3f12c8723022ef90bf2520442
I20260620 18:42:02.123342 26656 consensus_queue.cc:260] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } }
I20260620 18:42:02.123436 26656 raft_consensus.cc:388] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:02.123474 26656 raft_consensus.cc:482] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:02.123533 26656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:02.124780 26656 raft_consensus.cc:504] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } }
I20260620 18:42:02.124915 26656 leader_election.cc:302] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cae28bf3223a4903a5ee281684216b1f; no voters:
I20260620 18:42:02.125111 26656 leader_election.cc:288] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:42:02.125180 26661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:02.125392 26656 sys_catalog.cc:564] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:02.125427 26661 raft_consensus.cc:686] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 1 LEADER]: Becoming Leader. State: Replica: cae28bf3223a4903a5ee281684216b1f, State: Running, Role: LEADER
I20260620 18:42:02.125660 26661 consensus_queue.cc:237] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } }
I20260620 18:42:02.126420 26662 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cae28bf3223a4903a5ee281684216b1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } } }
I20260620 18:42:02.126578 26662 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:02.126931 26663 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae28bf3223a4903a5ee281684216b1f. Latest consensus state: current_term: 1 leader_uuid: "cae28bf3223a4903a5ee281684216b1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } } }
I20260620 18:42:02.127187 26663 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [sys.catalog]: This master's current role is: LEADER
W20260620 18:42:02.127527 26677 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:42:02.127591 26677 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:42:02.127650 26674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:02.127836 26674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:02.129693 26674 catalog_manager.cc:1347] Generated new cluster ID: 9c157558f6464ebebc10107d39aeea0b
I20260620 18:42:02.129760 26674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:42:02 dist-test-slave-wp6g krb5kdc[26575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:42:02.136807 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.123802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.10.250.126:44805)
0620 18:42:02.124061 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:02.124071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:02.124430 (+ 359us) server_negotiation.cc:407] Connection header received
0620 18:42:02.124583 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:02.124592 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:02.124657 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:02.124833 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:02.125881 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.127492 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.128604 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.129276 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.129404 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.132703 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:02.132709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:02.132709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:02.132724 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:02.134919 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:02.135414 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:02.135418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:02.135419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:02.135562 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:02.135792 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:02.135793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:02.135793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:02.136116 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:02.136254 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:02.136431 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 18:42:02.136588 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":106,"threads_started":1}
I20260620 18:42:02.144887 26674 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:42:02.145663 26674 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:02.156864 26674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Generated new TSK 0
I20260620 18:42:02.157147 26674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:42:02.190236 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44805
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:02.279739 26682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:02.280006 26682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:02.280049 26682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:02.281291 26682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:02.281371 26682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:02.281420 26682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:42:02.283095 26682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:42:02.283401 26682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:02.283661 26682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:02.289734 26687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:02.289733 26688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:02.289758 26690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:02.289733 26682 server_base.cc:1034] running on GCE node
I20260620 18:42:02.290217 26682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:02.290484 26682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:02.291697 26682 hybrid_clock.cc:648] HybridClock initialized: now 1781980922291675 us; error 46 us; skew 500 ppm
Jun 20 18:42:02 dist-test-slave-wp6g krb5kdc[26575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980922, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:02.294694 26682 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:42:02.295207 26682 webserver.cc:460] Webserver started at http://127.10.250.65:40779/ using document root <none> and password file <none>
I20260620 18:42:02.295499 26682 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:02.295573 26682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:02.295730 26682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:02.297219 26682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "09e1ea8f3b2243e1be7127c62a05ed49"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "60f545429e3fa66fe6ddd29076d927ad"
server_key_iv: "60d9c72acc85d964fc9353576035cdb1"
server_key_version: "encryptionkey@0"
I20260620 18:42:02.297811 26682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "09e1ea8f3b2243e1be7127c62a05ed49"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "60f545429e3fa66fe6ddd29076d927ad"
server_key_iv: "60d9c72acc85d964fc9353576035cdb1"
server_key_version: "encryptionkey@0"
I20260620 18:42:02.299422 26682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260620 18:42:02.300447 26697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:02.300688 26682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:42:02.300822 26682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "09e1ea8f3b2243e1be7127c62a05ed49"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "60f545429e3fa66fe6ddd29076d927ad"
server_key_iv: "60d9c72acc85d964fc9353576035cdb1"
server_key_version: "encryptionkey@0"
I20260620 18:42:02.300912 26682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:02.310498 26682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:02.312888 26682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:02.313076 26682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:02.313323 26682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:02.313649 26682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:02.313702 26682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:02.313736 26682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:02.313766 26682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:02.320186 26682 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:36959
I20260620 18:42:02.320240 26810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:36959 every 8 connection(s)
I20260620 18:42:02.320633 26682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:42:02.326059 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 26682
I20260620 18:42:02.326171 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:42:02.326370 11241 external_mini_cluster.cc:1428] Setting key 4adf6f68b4158c45ccf7f8ba5cf30d87
Jun 20 18:42:02 dist-test-slave-wp6g krb5kdc[26575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980922, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:42:02.332609 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.321613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:39901 (local address 127.10.250.126:44805)
0620 18:42:02.321664 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:02.321666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:02.321855 (+ 189us) server_negotiation.cc:407] Connection header received
0620 18:42:02.322389 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:02.322390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:02.322400 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:02.322470 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:02.323594 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.324136 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.325332 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.325632 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.325849 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.328488 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:02.328490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:02.328491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:02.328504 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:02.330607 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:02.331262 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:02.331264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:02.331264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:02.331335 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:02.331721 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:02.331722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:02.331722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:02.331902 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:02.331971 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:02.332433 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 18:42:02.332534 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:42:02.332918 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.321537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:39901)
0620 18:42:02.321708 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:02.321762 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:02.322252 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:02.322613 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:02.322617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:02.323258 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:02.323470 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:02.323476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.324267 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.324268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:02.325190 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:02.325193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.325749 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.325750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:02.325903 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.326628 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:02.326641 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:02.328376 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:02.330712 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:02.330726 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:02.330729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:02.331153 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:02.331428 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:02.331429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:02.331430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:02.331617 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:02.331989 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:02.331997 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:02.332219 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 18:42:02.332362 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 18:42:02.332586 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260620 18:42:02.333542 26811 heartbeater.cc:347] Connected to a master server at 127.10.250.126:44805
I20260620 18:42:02.333735 26811 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:02.333959 26811 heartbeater.cc:511] Master 127.10.250.126:44805 requested a full tablet report, sending...
I20260620 18:42:02.334709 26620 ts_manager.cc:194] Registered new tserver with Master: 09e1ea8f3b2243e1be7127c62a05ed49 (127.10.250.65:36959)
I20260620 18:42:02.335471 26620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:39901
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:42:02.359457 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44805
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:02.449051 26816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:02.449318 26816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:02.449350 26816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:02.450733 26816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:02.450795 26816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:02.450827 26816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:42:02.452520 26816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:42:02.452798 26816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:02.453076 26816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:02.458874 26824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:02.458879 26822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:02.458874 26821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:02.459322 26816 server_base.cc:1034] running on GCE node
I20260620 18:42:02.459554 26816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:02.459889 26816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:02.461098 26816 hybrid_clock.cc:648] HybridClock initialized: now 1781980922461083 us; error 40 us; skew 500 ppm
Jun 20 18:42:02 dist-test-slave-wp6g krb5kdc[26575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980922, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:02.464172 26816 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:42:02.464882 26816 webserver.cc:460] Webserver started at http://127.10.250.66:41165/ using document root <none> and password file <none>
I20260620 18:42:02.465188 26816 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:02.465262 26816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:02.465430 26816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:02.466758 26816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "f135ed6d060eed10a1cbc13c449594ba"
server_key_iv: "5bd14d5dac5d803ef139d15ed9354416"
server_key_version: "encryptionkey@0"
I20260620 18:42:02.467183 26816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "f135ed6d060eed10a1cbc13c449594ba"
server_key_iv: "5bd14d5dac5d803ef139d15ed9354416"
server_key_version: "encryptionkey@0"
I20260620 18:42:02.468716 26816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 18:42:02.469659 26831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:02.470075 26816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:42:02.470211 26816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "f135ed6d060eed10a1cbc13c449594ba"
server_key_iv: "5bd14d5dac5d803ef139d15ed9354416"
server_key_version: "encryptionkey@0"
I20260620 18:42:02.470301 26816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:02.500630 26816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:02.503720 26816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:02.503925 26816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:02.504215 26816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:02.504592 26816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:02.504652 26816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:02.504832 26816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:02.504885 26816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:02.511907 26816 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:46653
I20260620 18:42:02.511948 26944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:46653 every 8 connection(s)
I20260620 18:42:02.512311 26816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:42:02.515635 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 26816
I20260620 18:42:02.515740 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:42:02.515933 11241 external_mini_cluster.cc:1428] Setting key db1fc7472c24c73a8be1eb166ebfbe90
Jun 20 18:42:02 dist-test-slave-wp6g krb5kdc[26575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980922, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:42:02.523522 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.513134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:50403 (local address 127.10.250.126:44805)
0620 18:42:02.513200 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:02.513202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:02.513446 (+ 244us) server_negotiation.cc:407] Connection header received
0620 18:42:02.514025 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:02.514026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:02.514039 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:02.514087 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:02.515182 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.515950 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.517270 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.517602 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.517687 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.520077 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:02.520079 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:02.520079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:02.520089 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:02.521870 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:02.522394 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:02.522396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:02.522396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:02.522469 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:02.522796 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:02.522797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:02.522798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:02.522937 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:02.523018 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:02.523326 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 18:42:02.523425 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:42:02.523749 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.513117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:50403)
0620 18:42:02.513314 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:02.513353 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:02.513895 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:02.514172 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:02.514175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:02.514858 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:02.515068 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:02.515074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.516098 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.516100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:02.517167 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:02.517170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.517701 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.517701 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:02.517820 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.518419 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:02.518429 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:02.519969 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:02.521999 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:02.522010 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:02.522012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:02.522302 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:02.522581 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:02.522582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:02.522582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:02.522720 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:02.523008 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:02.523016 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:02.523179 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 18:42:02.523292 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:42:02.523483 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
I20260620 18:42:02.524317 26945 heartbeater.cc:347] Connected to a master server at 127.10.250.126:44805
I20260620 18:42:02.524447 26945 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:02.524664 26945 heartbeater.cc:511] Master 127.10.250.126:44805 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
I20260620 18:42:02.525103 26620 ts_manager.cc:194] Registered new tserver with Master: 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653)
I20260620 18:42:02.525523 26620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:50403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:42:02.547859 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44805
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:02.637916 26950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:02.638151 26950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:02.638178 26950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:02.639397 26950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:02.639446 26950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:02.639472 26950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:42:02.640931 26950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:42:02.641142 26950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:02.641328 26950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:02.646699 26956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:02.646720 26955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:02.646867 26950 server_base.cc:1034] running on GCE node
W20260620 18:42:02.646720 26958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:02.647279 26950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:02.647552 26950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:02.648729 26950 hybrid_clock.cc:648] HybridClock initialized: now 1781980922648713 us; error 28 us; skew 500 ppm
Jun 20 18:42:02 dist-test-slave-wp6g krb5kdc[26575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980922, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:02.658277 26950 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:42:02.658792 26950 webserver.cc:460] Webserver started at http://127.10.250.67:41079/ using document root <none> and password file <none>
I20260620 18:42:02.659039 26950 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:02.659086 26950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:02.659273 26950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:02.660398 26950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "91a20fac9dff44e2a7d71ff3a22939c7"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "6c4307d473c2316c3f431c47996656b7"
server_key_iv: "66d70f9b23f261b36e3223fa13b825bc"
server_key_version: "encryptionkey@0"
I20260620 18:42:02.660773 26950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "91a20fac9dff44e2a7d71ff3a22939c7"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "6c4307d473c2316c3f431c47996656b7"
server_key_iv: "66d70f9b23f261b36e3223fa13b825bc"
server_key_version: "encryptionkey@0"
I20260620 18:42:02.662307 26950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:42:02.663389 26965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:02.663633 26950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:42:02.663743 26950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "91a20fac9dff44e2a7d71ff3a22939c7"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "6c4307d473c2316c3f431c47996656b7"
server_key_iv: "66d70f9b23f261b36e3223fa13b825bc"
server_key_version: "encryptionkey@0"
I20260620 18:42:02.663851 26950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:02.679059 26950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:02.681646 26950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:02.681831 26950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:02.682097 26950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:02.682440 26950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:02.682497 26950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:02.682557 26950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:02.682590 26950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:02.688781 26950 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:36547
I20260620 18:42:02.688880 27078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:36547 every 8 connection(s)
I20260620 18:42:02.689208 26950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:42:02.693682 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 26950
I20260620 18:42:02.693770 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:42:02.693964 11241 external_mini_cluster.cc:1428] Setting key 46692dfe59e81b461569366db34c7c9d
Jun 20 18:42:02 dist-test-slave-wp6g krb5kdc[26575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980922, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:42:02.699303 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.690018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:48461 (local address 127.10.250.126:44805)
0620 18:42:02.690075 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:02.690077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:02.690313 (+ 236us) server_negotiation.cc:407] Connection header received
0620 18:42:02.690947 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:02.690948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:02.690960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:02.691017 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:02.691963 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.692446 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.693435 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.693776 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.693910 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.696455 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:02.696457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:02.696458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:02.696465 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:02.697791 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:02.698233 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:02.698235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:02.698235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:02.698302 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:02.698588 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:02.698589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:02.698590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:02.698721 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:02.698804 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:02.699120 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 18:42:02.699208 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 18:42:02.699527 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.690004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:48461)
0620 18:42:02.690176 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:02.690213 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:02.690840 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:02.691081 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:02.691085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:02.691675 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:02.691882 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:02.691886 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.692524 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.692525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:02.693344 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:02.693346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.693924 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.693925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:02.694062 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.694781 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:02.694793 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:02.696356 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:02.697901 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:02.697909 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:02.697912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:02.698151 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:02.698376 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:02.698377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:02.698377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:02.698497 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:02.698792 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:02.698799 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:02.698961 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 18:42:02.699086 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 18:42:02.699267 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260620 18:42:02.700084 27079 heartbeater.cc:347] Connected to a master server at 127.10.250.126:44805
I20260620 18:42:02.700193 27079 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:02.700379 27079 heartbeater.cc:511] Master 127.10.250.126:44805 requested a full tablet report, sending...
I20260620 18:42:02.700747 26620 ts_manager.cc:194] Registered new tserver with Master: 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67:36547)
I20260620 18:42:02.701099 26620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:48461
I20260620 18:42:02.708886 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:42:02.721333 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.710667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.10.250.126:44805)
0620 18:42:02.710724 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:02.710726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:02.710972 (+ 246us) server_negotiation.cc:407] Connection header received
0620 18:42:02.711097 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:02.711097 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:02.711109 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:02.711194 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:02.712053 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.712759 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.713823 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.714158 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.714222 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.718036 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:02.718038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:02.718038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:02.718047 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:02.720021 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:02.720446 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:02.720448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:02.720448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:02.720511 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:02.720722 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:02.720723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:02.720724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:02.720903 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:02.721014 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:02.721121 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 18:42:02.721212 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:42:02.724478 26620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:42:02.727320 26620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:42:02.735109 27090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.730187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:36959 (local address 127.0.0.1:34596)
0620 18:42:02.730293 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:02.730336 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:02.730402 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:02.731492 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:02.731493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:02.731495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:02.731533 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:02.731535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.732755 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.732756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:02.733871 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:02.733873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.734769 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.734770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:02.734871 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.734887 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 18:42:02.734937 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 18:42:02.735004 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":7,"thread_start_us":50,"threads_started":1}
I20260620 18:42:02.735186 27094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.730740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.10.250.67:36547)
0620 18:42:02.730931 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:02.730935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:02.730946 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:42:02.730987 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:02.730989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:02.731005 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:02.731080 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:02.731325 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.732449 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.733868 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.734664 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.734740 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.734809 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:02.735050 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 18:42:02.735093 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
I20260620 18:42:02.735121 27091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.730212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:46653 (local address 127.0.0.1:46532)
0620 18:42:02.730348 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:02.730360 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:02.730425 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:02.730857 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:02.730860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:02.730862 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:02.730951 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:02.730953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.732086 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.732087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:02.733773 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:02.733777 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.734857 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.734859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:02.734926 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.734932 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:42:02.734968 (+ 36us) client_negotiation.cc:240] Negotiation successful
0620 18:42:02.735023 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":93,"threads_started":1}
I20260620 18:42:02.735415 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.730398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:36547 (local address 127.0.0.1:48574)
0620 18:42:02.730673 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:02.730684 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:02.730732 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:02.731166 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:02.731167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:02.731168 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:02.731228 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:02.731230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.732558 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.732559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:02.733768 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:02.733771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.734840 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.734840 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:02.734892 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.734895 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:42:02.734955 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 18:42:02.735004 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":35,"thread_start_us":94,"threads_started":1}
I20260620 18:42:02.735707 27095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.730750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.10.250.65:36959)
0620 18:42:02.731298 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:02.731303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:02.731314 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:42:02.731358 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:02.731360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:02.731376 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:02.731473 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:02.731609 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.732645 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.733962 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.734669 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.735474 (+ 805us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.735548 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:02.735578 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 18:42:02.735623 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":110,"threads_started":1}
I20260620 18:42:02.735757 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.730469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.10.250.66:46653)
0620 18:42:02.730658 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:02.730664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:02.730675 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:42:02.730719 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:02.730722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:02.730750 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:02.731037 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:02.731115 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.731986 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.733882 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.734757 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.735163 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.735584 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:02.735617 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 18:42:02.735673 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":111,"threads_started":1}
I20260620 18:42:02.736198 27013 tablet_service.cc:1467] Processing CreateTablet for tablet 093d2d87920e45a6b7c8d39d360e1524 (DEFAULT_TABLE table=test-table [id=6db70714525646d39196132ce3935f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:02.736356 26745 tablet_service.cc:1467] Processing CreateTablet for tablet 093d2d87920e45a6b7c8d39d360e1524 (DEFAULT_TABLE table=test-table [id=6db70714525646d39196132ce3935f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:02.736364 26879 tablet_service.cc:1467] Processing CreateTablet for tablet 093d2d87920e45a6b7c8d39d360e1524 (DEFAULT_TABLE table=test-table [id=6db70714525646d39196132ce3935f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:02.736503 27013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093d2d87920e45a6b7c8d39d360e1524. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:02.736687 26745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093d2d87920e45a6b7c8d39d360e1524. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:02.736687 26879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093d2d87920e45a6b7c8d39d360e1524. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:02.739331 27096 tablet_bootstrap.cc:492] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7: Bootstrap starting.
I20260620 18:42:02.740079 27098 tablet_bootstrap.cc:492] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Bootstrap starting.
I20260620 18:42:02.740223 27096 tablet_bootstrap.cc:654] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:02.740823 27098 tablet_bootstrap.cc:654] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:02.740854 27096 log.cc:824] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:02.741367 27098 log.cc:824] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:02.741739 27096 tablet_bootstrap.cc:492] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7: No bootstrap required, opened a new log
I20260620 18:42:02.741755 27097 tablet_bootstrap.cc:492] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49: Bootstrap starting.
I20260620 18:42:02.741845 27096 ts_tablet_manager.cc:1397] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:42:02.742406 27097 tablet_bootstrap.cc:654] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:02.743013 27097 log.cc:824] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:02.743893 27097 tablet_bootstrap.cc:492] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49: No bootstrap required, opened a new log
I20260620 18:42:02.743973 27097 ts_tablet_manager.cc:1397] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:42:02.744194 27096 raft_consensus.cc:348] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } } peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } }
I20260620 18:42:02.744396 27096 raft_consensus.cc:374] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:02.744431 27096 raft_consensus.cc:729] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a20fac9dff44e2a7d71ff3a22939c7, State: Initialized, Role: FOLLOWER
I20260620 18:42:02.744580 27096 consensus_queue.cc:260] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } } peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } }
I20260620 18:42:02.744874 27096 ts_tablet_manager.cc:1428] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:02.744925 27079 heartbeater.cc:503] Master 127.10.250.126:44805 was elected leader, sending a full tablet report...
I20260620 18:42:02.745594 27098 tablet_bootstrap.cc:492] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: No bootstrap required, opened a new log
I20260620 18:42:02.745669 27098 ts_tablet_manager.cc:1397] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 18:42:02.745951 27097 raft_consensus.cc:348] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } } peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } }
I20260620 18:42:02.746147 27097 raft_consensus.cc:374] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:02.746192 27097 raft_consensus.cc:729] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e1ea8f3b2243e1be7127c62a05ed49, State: Initialized, Role: FOLLOWER
I20260620 18:42:02.746326 27097 consensus_queue.cc:260] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } } peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } }
I20260620 18:42:02.746661 26811 heartbeater.cc:503] Master 127.10.250.126:44805 was elected leader, sending a full tablet report...
I20260620 18:42:02.746662 27097 ts_tablet_manager.cc:1428] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 18:42:02.747987 27098 raft_consensus.cc:348] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } } peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } }
I20260620 18:42:02.748152 27098 raft_consensus.cc:374] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:02.748193 27098 raft_consensus.cc:729] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e942e3f2c6c4691aa2b67b9b3e4dcfb, State: Initialized, Role: FOLLOWER
I20260620 18:42:02.748311 27098 consensus_queue.cc:260] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } } peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } }
I20260620 18:42:02.748594 27098 ts_tablet_manager.cc:1428] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:02.748680 26945 heartbeater.cc:503] Master 127.10.250.126:44805 was elected leader, sending a full tablet report...
W20260620 18:42:02.762952 26946 tablet.cc:2369] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:42:02.791169 27102 raft_consensus.cc:482] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:02.791311 27102 raft_consensus.cc:504] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } } peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } }
I20260620 18:42:02.791756 27102 leader_election.cc:288] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09e1ea8f3b2243e1be7127c62a05ed49 (127.10.250.65:36959), 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653)
I20260620 18:42:02.795917 27095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.792194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:59501 (local address 127.10.250.65:36959)
0620 18:42:02.792285 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:02.792287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:02.792586 (+ 299us) server_negotiation.cc:407] Connection header received
0620 18:42:02.792617 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:02.792618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:02.792629 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:02.792692 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:02.792810 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.793545 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.794910 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.795504 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.795584 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.795593 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:02.795790 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 18:42:02.795841 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:42:02.795917 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.792112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:33979 (local address 127.10.250.66:46653)
0620 18:42:02.792213 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:02.792215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:02.792228 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:42:02.792263 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:02.792264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:02.792277 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:02.792355 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:02.792584 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.793410 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.794910 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.795503 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.795584 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.795593 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:02.795790 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 18:42:02.795841 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:42:02.795939 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.792004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:46653 (local address 127.10.250.67:33979)
0620 18:42:02.792080 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:02.792095 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:02.792157 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:02.792370 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:02.792371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:02.792373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:02.792472 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:02.792474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.793492 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.793492 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:02.794813 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:02.794815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.795598 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.795599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:02.795683 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.795685 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:42:02.795750 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 18:42:02.795852 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":37}
I20260620 18:42:02.796216 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.792004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:36959 (local address 127.10.250.67:59501)
0620 18:42:02.792458 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:02.792468 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:02.792529 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:02.792695 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:02.792695 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:02.792696 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:02.792747 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:02.792748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.793617 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.793618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:02.794813 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:02.794815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.795598 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.795599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:02.795680 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.795684 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:42:02.795750 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 18:42:02.795851 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":279,"thread_start_us":156,"threads_started":1}
I20260620 18:42:02.796350 26899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093d2d87920e45a6b7c8d39d360e1524" candidate_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" is_pre_election: true
I20260620 18:42:02.796555 26899 raft_consensus.cc:2454] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91a20fac9dff44e2a7d71ff3a22939c7 in term 0.
I20260620 18:42:02.796537 26765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093d2d87920e45a6b7c8d39d360e1524" candidate_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" is_pre_election: true
I20260620 18:42:02.796751 26765 raft_consensus.cc:2454] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91a20fac9dff44e2a7d71ff3a22939c7 in term 0.
I20260620 18:42:02.796825 26967 leader_election.cc:302] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e942e3f2c6c4691aa2b67b9b3e4dcfb, 91a20fac9dff44e2a7d71ff3a22939c7; no voters:
I20260620 18:42:02.797204 27102 raft_consensus.cc:2781] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:42:02.797260 27102 raft_consensus.cc:482] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:42:02.797279 27102 raft_consensus.cc:3037] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:02.798031 27102 raft_consensus.cc:504] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } } peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } }
I20260620 18:42:02.798184 27102 leader_election.cc:288] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [CANDIDATE]: Term 1 election: Requested vote from peers 09e1ea8f3b2243e1be7127c62a05ed49 (127.10.250.65:36959), 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653)
I20260620 18:42:02.798348 26765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093d2d87920e45a6b7c8d39d360e1524" candidate_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e1ea8f3b2243e1be7127c62a05ed49"
I20260620 18:42:02.798413 26899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093d2d87920e45a6b7c8d39d360e1524" candidate_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb"
I20260620 18:42:02.798435 26765 raft_consensus.cc:3037] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:02.798486 26899 raft_consensus.cc:3037] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:02.799407 26765 raft_consensus.cc:2454] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91a20fac9dff44e2a7d71ff3a22939c7 in term 1.
I20260620 18:42:02.799407 26899 raft_consensus.cc:2454] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91a20fac9dff44e2a7d71ff3a22939c7 in term 1.
I20260620 18:42:02.799603 26967 leader_election.cc:302] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e942e3f2c6c4691aa2b67b9b3e4dcfb, 91a20fac9dff44e2a7d71ff3a22939c7; no voters:
I20260620 18:42:02.799727 27102 raft_consensus.cc:2781] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:02.799921 27102 raft_consensus.cc:686] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 1 LEADER]: Becoming Leader. State: Replica: 91a20fac9dff44e2a7d71ff3a22939c7, State: Running, Role: LEADER
I20260620 18:42:02.800182 27102 consensus_queue.cc:237] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } } peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } }
I20260620 18:42:02.801108 26620 catalog_manager.cc:5495] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67). New cstate: current_term: 1 leader_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } health_report { overall_health: UNKNOWN } } }
I20260620 18:42:02.808099 27094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.805523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.10.250.67:36547)
0620 18:42:02.805592 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:02.805594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:02.805602 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:42:02.805634 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:02.805635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:02.805653 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:02.805733 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:42:02.805890 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.806363 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.807225 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:02.807440 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:02.807509 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:02.807722 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:42:02.807841 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:42:02.807928 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:02.808012 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 18:42:02.808035 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:42:02.810195 26765 raft_consensus.cc:1261] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Refusing update from remote peer 91a20fac9dff44e2a7d71ff3a22939c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:02.810209 26899 raft_consensus.cc:1261] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 1 FOLLOWER]: Refusing update from remote peer 91a20fac9dff44e2a7d71ff3a22939c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:02.810673 27102 consensus_queue.cc:1035] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:02.810873 27106 consensus_queue.cc:1035] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:02.813323 27109 mvcc.cc:205] Tried to move back new op lower bound from 7298993859827802112 to 7298993859790417920. Current Snapshot: MvccSnapshot[applied={T|T < 7298993859827802112}]
I20260620 18:42:02.814210 27110 mvcc.cc:205] Tried to move back new op lower bound from 7298993859827802112 to 7298993859790417920. Current Snapshot: MvccSnapshot[applied={T|T < 7298993859827802112}]
I20260620 18:42:02.815939 26619 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:02.816056 26619 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:02.816985 26619 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Sending DeleteTablet for 3 replicas of tablet 093d2d87920e45a6b7c8d39d360e1524
I20260620 18:42:02.817370 26745 tablet_service.cc:1514] Processing DeleteTablet for tablet 093d2d87920e45a6b7c8d39d360e1524 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:02 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:34596
I20260620 18:42:02.817389 27013 tablet_service.cc:1514] Processing DeleteTablet for tablet 093d2d87920e45a6b7c8d39d360e1524 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:02 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:48574
I20260620 18:42:02.817529 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 26682
I20260620 18:42:02.817629 26879 tablet_service.cc:1514] Processing DeleteTablet for tablet 093d2d87920e45a6b7c8d39d360e1524 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:02 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:46532
I20260620 18:42:02.818660 27119 tablet_replica.cc:330] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7: stopping tablet replica
I20260620 18:42:02.818784 27119 raft_consensus.cc:2229] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:42:02.818912 27119 raft_consensus.cc:2258] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:02.819319 27119 ts_tablet_manager.cc:1905] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:02.821103 27119 ts_tablet_manager.cc:1918] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:02.821177 27119 log.cc:1196] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/093d2d87920e45a6b7c8d39d360e1524
I20260620 18:42:02.821486 27119 ts_tablet_manager.cc:1939] T 093d2d87920e45a6b7c8d39d360e1524 P 91a20fac9dff44e2a7d71ff3a22939c7: Deleting consensus metadata
I20260620 18:42:02.821947 26607 catalog_manager.cc:4841] TS 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67:36547): tablet 093d2d87920e45a6b7c8d39d360e1524 (table test-table [id=6db70714525646d39196132ce3935f85]) successfully deleted
W20260620 18:42:02.823488 26607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:36959 (error 108)
I20260620 18:42:02.823573 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 26816
I20260620 18:42:02.823596 27120 tablet_replica.cc:330] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: stopping tablet replica
I20260620 18:42:02.823755 27120 raft_consensus.cc:2229] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:02.823889 27120 raft_consensus.cc:2258] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:02.824633 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.824478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:36959 (local address 127.0.0.1:34612)
0620 18:42:02.824534 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:02.824587 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 18:42:02.824783 26607 catalog_manager.cc:4584] TS 09e1ea8f3b2243e1be7127c62a05ed49 (127.10.250.65:36959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 093d2d87920e45a6b7c8d39d360e1524: Network error: Client connection negotiation failed: client connection to 127.10.250.65:36959: connect: Connection refused (error 111)
I20260620 18:42:02.830446 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 26950
I20260620 18:42:02.830621 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:02.830179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:46653 (local address 127.0.0.1:46546)
0620 18:42:02.830539 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:02.830575 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.66:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318}
I20260620 18:42:02.835428 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 26588
I20260620 18:42:22.841033 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44805
--webserver_interface=127.10.250.126
--webserver_port=44125
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:44805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:22.938648 27122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:22.938905 27122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:22.938932 27122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:22.940275 27122 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:42:22.940327 27122 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:42:22.940341 27122 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:42:22.940353 27122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:22.940368 27122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:22.940382 27122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:22.940393 27122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:22.940405 27122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:22.942010 27122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:44805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=44125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:42:22.942238 27122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:22.942433 27122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:22.952445 27129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:22.952432 27128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:22.952489 27122 server_base.cc:1034] running on GCE node
W20260620 18:42:22.952432 27131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:22.953226 27122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:22.953565 27122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:22.954708 27122 hybrid_clock.cc:648] HybridClock initialized: now 1781980942954668 us; error 52 us; skew 500 ppm
Jun 20 18:42:22 dist-test-slave-wp6g krb5kdc[26575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980942, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:22.957715 27122 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:42:22.958339 27122 webserver.cc:460] Webserver started at http://127.10.250.126:44125/ using document root <none> and password file <none>
I20260620 18:42:22.958820 27122 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:22.958894 27122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:22.960647 27122 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:42:22.961539 27138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:22.961815 27122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:42:22.962090 27122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "cae28bf3223a4903a5ee281684216b1f"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "88be9f89db06ad092804d321d8782e68"
server_key_iv: "1ae82d7d2a8c3007e1e9ca2f8ab02d24"
server_key_version: "encryptionkey@0"
I20260620 18:42:22.962678 27122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:22.979624 27122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:22.981971 27122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:22.982139 27122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:22.986904 27122 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:44805
I20260620 18:42:22.986979 27190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:44805 every 8 connection(s)
I20260620 18:42:22.987378 27122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:22.989035 27191 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:42:22.989787 27191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Bootstrap starting.
I20260620 18:42:22.997057 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 27122
I20260620 18:42:22.997426 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:36959
--local_ip_for_outbound_sockets=127.10.250.65
--tserver_master_addrs=127.10.250.126:44805
--webserver_port=40779
--webserver_interface=127.10.250.65
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:42:22.999219 27191 log.cc:824] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:23.001824 27191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:42:23.002234 27191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Bootstrap complete.
I20260620 18:42:23.004755 27191 raft_consensus.cc:348] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } }
I20260620 18:42:23.005004 27191 raft_consensus.cc:729] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae28bf3223a4903a5ee281684216b1f, State: Initialized, Role: FOLLOWER
I20260620 18:42:23.005189 27191 consensus_queue.cc:260] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } }
I20260620 18:42:23.005264 27191 raft_consensus.cc:388] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:23.005338 27191 raft_consensus.cc:482] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:23.005403 27191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:42:23.006606 27191 raft_consensus.cc:504] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } }
I20260620 18:42:23.006701 27191 leader_election.cc:302] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cae28bf3223a4903a5ee281684216b1f; no voters:
I20260620 18:42:23.006922 27191 leader_election.cc:288] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:42:23.006964 27195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:42:23.007097 27195 raft_consensus.cc:686] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 2 LEADER]: Becoming Leader. State: Replica: cae28bf3223a4903a5ee281684216b1f, State: Running, Role: LEADER
I20260620 18:42:23.007185 27195 consensus_queue.cc:237] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } }
I20260620 18:42:23.007213 27191 sys_catalog.cc:564] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:23.007490 27195 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae28bf3223a4903a5ee281684216b1f. Latest consensus state: current_term: 2 leader_uuid: "cae28bf3223a4903a5ee281684216b1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } } }
I20260620 18:42:23.007562 27195 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:23.007990 27197 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cae28bf3223a4903a5ee281684216b1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } } }
I20260620 18:42:23.008072 27197 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:23.008436 27199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:23.009423 27199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:23.009536 27199 catalog_manager.cc:1259] Loaded cluster ID: 9c157558f6464ebebc10107d39aeea0b
I20260620 18:42:23.009572 27199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:23.010850 27199 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:23.011075 27199 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Loaded TSK: 0
I20260620 18:42:23.021548 27199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Generated new TSK 1
I20260620 18:42:23.021677 27199 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:42:23.091141 27193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:23.091365 27193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:23.091391 27193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:23.092638 27193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:23.092681 27193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:23.092708 27193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:42:23.094209 27193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:36959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=40779
--tserver_master_addrs=127.10.250.126:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:42:23.094481 27193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:23.094766 27193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:23.100061 27218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:23.100137 27221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:23.100075 27219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:23.100466 27193 server_base.cc:1034] running on GCE node
I20260620 18:42:23.100710 27193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:23.101017 27193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:23.102190 27193 hybrid_clock.cc:648] HybridClock initialized: now 1781980943102164 us; error 39 us; skew 500 ppm
Jun 20 18:42:23 dist-test-slave-wp6g krb5kdc[26575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:23.104789 27193 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:42:23.105305 27193 webserver.cc:460] Webserver started at http://127.10.250.65:40779/ using document root <none> and password file <none>
I20260620 18:42:23.105557 27193 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:23.105613 27193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:23.107054 27193 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:42:23.107766 27228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:23.107952 27193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:42:23.108063 27193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "09e1ea8f3b2243e1be7127c62a05ed49"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "60f545429e3fa66fe6ddd29076d927ad"
server_key_iv: "60d9c72acc85d964fc9353576035cdb1"
server_key_version: "encryptionkey@0"
I20260620 18:42:23.108433 27193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:23.127671 27193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:23.130168 27193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:23.130290 27193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:23.130641 27193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:23.131191 27235 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 18:42:23.132712 27193 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 18:42:23.132763 27193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 18:42:23.132802 27193 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 18:42:23.133503 27193 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 18:42:23.133549 27193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 18:42:23.133598 27235 tablet_bootstrap.cc:492] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49: Bootstrap starting.
I20260620 18:42:23.141450 27193 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:36959
I20260620 18:42:23.141618 27342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:36959 every 8 connection(s)
I20260620 18:42:23.141995 27193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:42:23.143622 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 27193
I20260620 18:42:23.143980 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:46653
--local_ip_for_outbound_sockets=127.10.250.66
--tserver_master_addrs=127.10.250.126:44805
--webserver_port=41165
--webserver_interface=127.10.250.66
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:42:23.145783 27235 log.cc:824] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:23.148247 27235 tablet_bootstrap.cc:492] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:42:23.148586 27235 tablet_bootstrap.cc:492] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49: Bootstrap complete.
I20260620 18:42:23.149758 27235 ts_tablet_manager.cc:1397] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
Jun 20 18:42:23 dist-test-slave-wp6g krb5kdc[26575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:42:23.151163 27235 raft_consensus.cc:348] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } } peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } }
I20260620 18:42:23.151556 27235 raft_consensus.cc:729] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e1ea8f3b2243e1be7127c62a05ed49, State: Initialized, Role: FOLLOWER
I20260620 18:42:23.151722 27235 consensus_queue.cc:260] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } } peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } }
I20260620 18:42:23.152038 27235 ts_tablet_manager.cc:1428] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260620 18:42:23.156486 27346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.142954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:49233 (local address 127.10.250.126:44805)
0620 18:42:23.143126 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.143135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.143182 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:42:23.144166 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.144173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.144254 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.144423 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.145525 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.146942 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.148496 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.149144 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.149253 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.152287 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:23.152292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:23.152292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:23.152306 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:23.154362 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.154977 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.154981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.154981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.155105 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.155444 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.155445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.155445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.155751 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:23.155848 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.156227 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.156338 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260620 18:42:23.156770 27345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.142814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:49233)
0620 18:42:23.143004 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:23.143059 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:23.143817 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:23.144583 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:23.144588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:23.145053 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:23.145362 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.145370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.147071 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.147072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.148413 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.148416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.149251 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.149252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.149396 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.150004 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:23.150014 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:23.152142 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:23.154463 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:23.154472 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:23.154475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:23.154869 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:23.155191 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:23.155192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:23.155192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:23.155361 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:23.155842 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:23.155849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:23.156047 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 18:42:23.156180 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 18:42:23.156392 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
I20260620 18:42:23.157636 27343 heartbeater.cc:347] Connected to a master server at 127.10.250.126:44805
I20260620 18:42:23.157747 27343 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:23.157965 27343 heartbeater.cc:511] Master 127.10.250.126:44805 requested a full tablet report, sending...
I20260620 18:42:23.158820 27155 ts_manager.cc:194] Registered new tserver with Master: 09e1ea8f3b2243e1be7127c62a05ed49 (127.10.250.65:36959)
I20260620 18:42:23.160149 27155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:49233
I20260620 18:42:23.161227 27343 heartbeater.cc:503] Master 127.10.250.126:44805 was elected leader, sending a full tablet report...
Jun 20 18:42:23 dist-test-slave-wp6g krb5kdc[26575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980942, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for kudu/127.10.250.65@KRBTEST.COM
I20260620 18:42:23.168751 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.160441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.10.250.65:36959)
0620 18:42:23.160611 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.160615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.160626 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:42:23.160660 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.160662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.160677 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.160752 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.161271 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.162367 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.163214 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.163674 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.163746 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.165934 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:23.165941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:23.165941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:23.165947 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:23.167541 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.167978 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.167981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.167981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.168032 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.168234 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.168235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.168235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.168364 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:23.168453 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.168607 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.168684 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260620 18:42:23.168809 27350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.159511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:36959 (local address 127.0.0.1:45982)
0620 18:42:23.159690 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:23.159724 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:23.159787 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:23.160779 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:23.160785 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:23.161121 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:23.161182 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.161185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.162488 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.162489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.163135 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.163138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.163749 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.163750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.163806 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.164325 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:23.164329 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:23.165843 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:23.167620 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:23.167626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:23.167628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:23.167892 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:23.168096 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:23.168097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:23.168097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:23.168171 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:23.168429 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:23.168435 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:23.168489 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 18:42:23.168574 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 18:42:23.168706 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":119,"threads_started":1}
I20260620 18:42:23.169312 27277 tablet_service.cc:1514] Processing DeleteTablet for tablet 093d2d87920e45a6b7c8d39d360e1524 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:02 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:45982
I20260620 18:42:23.169888 27352 tablet_replica.cc:330] stopping tablet replica
I20260620 18:42:23.170008 27352 raft_consensus.cc:2229] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:23.170081 27352 raft_consensus.cc:2258] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:23.170404 27352 ts_tablet_manager.cc:1905] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:23.173179 27352 ts_tablet_manager.cc:1918] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:23.173269 27352 log.cc:1196] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal/wals/093d2d87920e45a6b7c8d39d360e1524
I20260620 18:42:23.173501 27352 ts_tablet_manager.cc:1939] T 093d2d87920e45a6b7c8d39d360e1524 P 09e1ea8f3b2243e1be7127c62a05ed49: Deleting consensus metadata
I20260620 18:42:23.174109 27142 catalog_manager.cc:4841] TS 09e1ea8f3b2243e1be7127c62a05ed49 (127.10.250.65:36959): tablet 093d2d87920e45a6b7c8d39d360e1524 (table test-table [id=6db70714525646d39196132ce3935f85]) successfully deleted
W20260620 18:42:23.238168 27347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:23.238389 27347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:23.238422 27347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:23.239705 27347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:23.239755 27347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:23.239784 27347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:42:23.241184 27347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:46653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=41165
--tserver_master_addrs=127.10.250.126:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:42:23.241379 27347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:23.241565 27347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:23.246793 27357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:23.246798 27360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:23.247051 27347 server_base.cc:1034] running on GCE node
W20260620 18:42:23.246815 27358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:23.247373 27347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:23.247632 27347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:23.248826 27347 hybrid_clock.cc:648] HybridClock initialized: now 1781980943248809 us; error 27 us; skew 500 ppm
Jun 20 18:42:23 dist-test-slave-wp6g krb5kdc[26575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:23.251451 27347 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:42:23.251938 27347 webserver.cc:460] Webserver started at http://127.10.250.66:41165/ using document root <none> and password file <none>
I20260620 18:42:23.252177 27347 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:23.252261 27347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:23.253722 27347 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:42:23.254494 27367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:23.254738 27347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:42:23.254870 27347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "f135ed6d060eed10a1cbc13c449594ba"
server_key_iv: "5bd14d5dac5d803ef139d15ed9354416"
server_key_version: "encryptionkey@0"
I20260620 18:42:23.255249 27347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:23.269332 27347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:23.271476 27347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:23.271646 27347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:23.271898 27347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:23.272330 27374 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 18:42:23.273922 27347 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 18:42:23.274004 27347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 18:42:23.274067 27347 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 18:42:23.274861 27347 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 18:42:23.274919 27347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260620 18:42:23.274986 27374 tablet_bootstrap.cc:492] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Bootstrap starting.
I20260620 18:42:23.282205 27347 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:46653
I20260620 18:42:23.282300 27481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:46653 every 8 connection(s)
I20260620 18:42:23.282698 27347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:42:23.284421 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 27347
I20260620 18:42:23.284726 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:36547
--local_ip_for_outbound_sockets=127.10.250.67
--tserver_master_addrs=127.10.250.126:44805
--webserver_port=41079
--webserver_interface=127.10.250.67
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:42:23.286162 27374 log.cc:824] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:23.288578 27374 tablet_bootstrap.cc:492] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:42:23.288954 27374 tablet_bootstrap.cc:492] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Bootstrap complete.
I20260620 18:42:23.290122 27374 ts_tablet_manager.cc:1397] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.014s
Jun 20 18:42:23 dist-test-slave-wp6g krb5kdc[26575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:42:23.292037 27374 raft_consensus.cc:348] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } } peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } }
I20260620 18:42:23.292496 27374 raft_consensus.cc:729] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e942e3f2c6c4691aa2b67b9b3e4dcfb, State: Initialized, Role: FOLLOWER
I20260620 18:42:23.292667 27374 consensus_queue.cc:260] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } } peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } }
I20260620 18:42:23.293016 27374 ts_tablet_manager.cc:1428] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 18:42:23.295876 27346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.283569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:60679 (local address 127.10.250.126:44805)
0620 18:42:23.283652 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.283655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.286677 (+ 3022us) server_negotiation.cc:407] Connection header received
0620 18:42:23.287239 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.287241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.287256 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.287313 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.288029 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.288536 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.289692 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.289907 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.289998 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.292409 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:23.292410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:23.292411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:23.292420 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:23.294311 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.294839 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.294841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.294841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.294907 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.295184 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.295185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.295185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.295313 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:23.295381 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.295682 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.295783 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 18:42:23.296303 27484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.283467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:60679)
0620 18:42:23.286463 (+ 2996us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:23.286544 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:23.287105 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:23.287435 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:23.287439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:23.287711 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:23.287920 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.287929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.288655 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.288656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.289596 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.289598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.290008 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.290009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.290143 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.290833 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:23.290852 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:23.292291 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:23.294444 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:23.294451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:23.294453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:23.294754 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:23.294985 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:23.294986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:23.294986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:23.295109 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:23.295383 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:23.295390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:23.295541 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 18:42:23.295649 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 18:42:23.295825 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2942,"thread_start_us":119,"threads_started":1}
I20260620 18:42:23.296828 27482 heartbeater.cc:347] Connected to a master server at 127.10.250.126:44805
I20260620 18:42:23.296947 27482 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:23.297137 27482 heartbeater.cc:511] Master 127.10.250.126:44805 requested a full tablet report, sending...
I20260620 18:42:23.297760 27155 ts_manager.cc:194] Registered new tserver with Master: 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653)
I20260620 18:42:23.298439 27155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:60679
I20260620 18:42:23.299698 27482 heartbeater.cc:503] Master 127.10.250.126:44805 was elected leader, sending a full tablet report...
Jun 20 18:42:23 dist-test-slave-wp6g krb5kdc[26575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980942, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for kudu/127.10.250.66@KRBTEST.COM
I20260620 18:42:23.306643 27488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.298143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49642 (local address 127.10.250.66:46653)
0620 18:42:23.298647 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.298653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.298663 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:42:23.298697 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.298699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.298713 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.298792 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.299119 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.299896 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.300942 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.301394 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.301461 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.303548 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:23.303563 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:23.303563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:23.303570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:23.305255 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.305669 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.305674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.305674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.305738 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.305953 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.305954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.305954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.306130 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:23.306219 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.306393 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.306490 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":110,"threads_started":1}
I20260620 18:42:23.306643 27350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.298089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:46653 (local address 127.0.0.1:49642)
0620 18:42:23.298143 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:23.298154 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:23.298195 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:23.298816 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:23.298817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:23.298992 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:23.299043 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.299044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.299975 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.299976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.300821 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.300825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.301470 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.301471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.301554 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.302024 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:23.302029 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:23.303443 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:23.305377 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:23.305378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:23.305379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:23.305579 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:23.305815 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:23.305816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:23.305817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:23.305884 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:23.306208 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:23.306209 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:23.306263 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 18:42:23.306355 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:42:23.306490 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 18:42:23.307236 27416 tablet_service.cc:1514] Processing DeleteTablet for tablet 093d2d87920e45a6b7c8d39d360e1524 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:02 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:49642
I20260620 18:42:23.307874 27489 tablet_replica.cc:330] stopping tablet replica
I20260620 18:42:23.308003 27489 raft_consensus.cc:2229] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:23.308080 27489 raft_consensus.cc:2258] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:23.308478 27489 ts_tablet_manager.cc:1905] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:23.311152 27489 ts_tablet_manager.cc:1918] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:23.311234 27489 log.cc:1196] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/093d2d87920e45a6b7c8d39d360e1524
I20260620 18:42:23.311452 27489 ts_tablet_manager.cc:1939] T 093d2d87920e45a6b7c8d39d360e1524 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Deleting consensus metadata
I20260620 18:42:23.312162 27140 catalog_manager.cc:4841] TS 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653): tablet 093d2d87920e45a6b7c8d39d360e1524 (table test-table [id=6db70714525646d39196132ce3935f85]) successfully deleted
W20260620 18:42:23.375628 27485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:23.375844 27485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:23.375869 27485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:23.377063 27485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:23.377108 27485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:23.377136 27485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:42:23.378741 27485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:36547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=41079
--tserver_master_addrs=127.10.250.126:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:42:23.379032 27485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:23.379285 27485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:23.385504 27494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:23.385418 27495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:23.385401 27485 server_base.cc:1034] running on GCE node
W20260620 18:42:23.385403 27497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:23.385891 27485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:23.386127 27485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:23.387319 27485 hybrid_clock.cc:648] HybridClock initialized: now 1781980943387307 us; error 27 us; skew 500 ppm
Jun 20 18:42:23 dist-test-slave-wp6g krb5kdc[26575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:23.390060 27485 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:42:23.390673 27485 webserver.cc:460] Webserver started at http://127.10.250.67:41079/ using document root <none> and password file <none>
I20260620 18:42:23.390951 27485 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:23.391059 27485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:23.392490 27485 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:42:23.393134 27504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:23.393374 27485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:42:23.393491 27485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "91a20fac9dff44e2a7d71ff3a22939c7"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "6c4307d473c2316c3f431c47996656b7"
server_key_iv: "66d70f9b23f261b36e3223fa13b825bc"
server_key_version: "encryptionkey@0"
I20260620 18:42:23.393862 27485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:23.418380 27485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:23.420478 27485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:23.420648 27485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:23.420913 27485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:23.421242 27485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:23.421298 27485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:23.421332 27485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:23.421363 27485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:23.427241 27485 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:36547
I20260620 18:42:23.427295 27617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:36547 every 8 connection(s)
I20260620 18:42:23.427676 27485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:42:23.432605 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 27485
Jun 20 18:42:23 dist-test-slave-wp6g krb5kdc[26575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:42:23.437894 27346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.428560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:39643 (local address 127.10.250.126:44805)
0620 18:42:23.428610 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.428613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.428800 (+ 187us) server_negotiation.cc:407] Connection header received
0620 18:42:23.429276 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.429278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.429289 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.429354 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.430346 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.430897 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.432103 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.432309 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.432371 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.434879 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:23.434881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:23.434881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:23.434889 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:23.436466 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.436904 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.436906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.436906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.436967 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.437222 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.437224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.437224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.437373 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:23.437457 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.437719 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.437825 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:42:23.438175 27620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.428477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:39643)
0620 18:42:23.428637 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:23.428681 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:23.429166 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:23.429431 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:23.429435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:23.430040 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:23.430241 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.430246 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.431003 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.431015 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.432017 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.432019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.432387 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.432388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.432495 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.433083 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:23.433093 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:23.434790 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:23.436521 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:23.436528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:23.436531 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:23.436815 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:23.437043 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:23.437044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:23.437044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:23.437153 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:23.437451 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:23.437456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:23.437578 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 18:42:23.437666 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:42:23.437880 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260620 18:42:23.438702 27618 heartbeater.cc:347] Connected to a master server at 127.10.250.126:44805
I20260620 18:42:23.438845 27618 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:23.439057 27618 heartbeater.cc:511] Master 127.10.250.126:44805 requested a full tablet report, sending...
I20260620 18:42:23.439455 27155 ts_manager.cc:194] Registered new tserver with Master: 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67:36547)
I20260620 18:42:23.439857 27155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:39643
I20260620 18:42:23.441039 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.433006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.10.250.126:44805)
0620 18:42:23.433156 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.433159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.433283 (+ 124us) server_negotiation.cc:407] Connection header received
0620 18:42:23.433361 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.433362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.433374 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.433462 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.434229 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.434858 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.435810 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.436155 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.436233 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.437739 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:23.437740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:23.437741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:23.437748 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:23.439702 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.440090 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.440091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.440091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.440144 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.440347 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.440348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.440348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.440474 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:23.440638 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.440764 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.440964 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260620 18:42:23.441866 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:42:23.445057 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.442231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.10.250.126:44805)
0620 18:42:23.442300 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.442301 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.442567 (+ 266us) server_negotiation.cc:407] Connection header received
0620 18:42:23.442609 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.442610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.442648 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.442712 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:42:23.442875 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.443486 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.444289 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.444552 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.444610 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.444789 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:42:23.444839 (+ 50us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 18:42:23.445006 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60136: token expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 18:42:23.445143 27621 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60136: token expired
W20260620 18:42:23.445257 27083 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.10.250.126:44805)
I20260620 18:42:23.451396 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.445563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60146 (local address 127.10.250.126:44805)
0620 18:42:23.445626 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.445629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.445684 (+ 55us) server_negotiation.cc:407] Connection header received
0620 18:42:23.445736 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.445737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.445744 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.445788 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.446275 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.446822 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.447579 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.447781 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.447835 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.449098 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:23.449099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:23.449099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:23.449106 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:23.450366 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.450799 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.450800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.450800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.450854 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.451020 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.451021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.451021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.451115 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:23.451186 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.451268 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.451331 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 18:42:23.452582 27155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:42:23.454708 27155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:42:23.457051 27416 tablet_service.cc:1467] Processing CreateTablet for tablet 01ef85a3f1f4452bb0cb0d016c6d78c9 (DEFAULT_TABLE table=test-table [id=a17d0dec48ce4ac59354e5d880efc787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:23.457052 27277 tablet_service.cc:1467] Processing CreateTablet for tablet 01ef85a3f1f4452bb0cb0d016c6d78c9 (DEFAULT_TABLE table=test-table [id=a17d0dec48ce4ac59354e5d880efc787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:23.457252 27277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ef85a3f1f4452bb0cb0d016c6d78c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:23.457252 27416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ef85a3f1f4452bb0cb0d016c6d78c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:23.458922 27235 tablet_bootstrap.cc:492] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49: Bootstrap starting.
I20260620 18:42:23.459221 27374 tablet_bootstrap.cc:492] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Bootstrap starting.
I20260620 18:42:23.459327 27235 tablet_bootstrap.cc:654] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:23.459559 27374 tablet_bootstrap.cc:654] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:23.460136 27235 tablet_bootstrap.cc:492] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49: No bootstrap required, opened a new log
I20260620 18:42:23.460189 27235 ts_tablet_manager.cc:1397] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:42:23.460489 27374 tablet_bootstrap.cc:492] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: No bootstrap required, opened a new log
I20260620 18:42:23.460529 27374 ts_tablet_manager.cc:1397] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:42:23.460575 27235 raft_consensus.cc:348] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } }
I20260620 18:42:23.460637 27235 raft_consensus.cc:374] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:23.460668 27235 raft_consensus.cc:729] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e1ea8f3b2243e1be7127c62a05ed49, State: Initialized, Role: FOLLOWER
I20260620 18:42:23.460660 27374 raft_consensus.cc:348] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } }
I20260620 18:42:23.460719 27374 raft_consensus.cc:374] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:23.460732 27235 consensus_queue.cc:260] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } }
I20260620 18:42:23.460762 27374 raft_consensus.cc:729] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e942e3f2c6c4691aa2b67b9b3e4dcfb, State: Initialized, Role: FOLLOWER
I20260620 18:42:23.460826 27235 ts_tablet_manager.cc:1428] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 18:42:23.460804 27374 consensus_queue.cc:260] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } }
I20260620 18:42:23.460911 27374 ts_tablet_manager.cc:1428] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:23.463117 27350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.456771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:36547 (local address 127.0.0.1:56636)
0620 18:42:23.456815 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:23.456826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:23.456863 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:23.459405 (+ 2542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:23.459406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:23.459408 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:23.459466 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.459468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.460931 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.460932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.462042 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.462045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.462825 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.462826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.462892 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.462918 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 18:42:23.462986 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 18:42:23.463039 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 18:42:23.463129 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.458965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56636 (local address 127.10.250.67:36547)
0620 18:42:23.459184 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.459189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.459203 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:42:23.459247 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.459249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.459266 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.459552 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:23.459645 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.460807 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.462131 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.462728 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.462824 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.462902 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.463002 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.463046 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":139,"threads_started":1}
I20260620 18:42:23.463878 27552 tablet_service.cc:1467] Processing CreateTablet for tablet 01ef85a3f1f4452bb0cb0d016c6d78c9 (DEFAULT_TABLE table=test-table [id=a17d0dec48ce4ac59354e5d880efc787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:23.464198 27552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ef85a3f1f4452bb0cb0d016c6d78c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:23.467103 27627 tablet_bootstrap.cc:492] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7: Bootstrap starting.
I20260620 18:42:23.467782 27627 tablet_bootstrap.cc:654] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:23.468413 27627 log.cc:824] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:23.469177 27627 tablet_bootstrap.cc:492] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7: No bootstrap required, opened a new log
I20260620 18:42:23.469262 27627 ts_tablet_manager.cc:1397] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:42:23.470709 27627 raft_consensus.cc:348] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } }
I20260620 18:42:23.470892 27627 raft_consensus.cc:374] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:23.470940 27627 raft_consensus.cc:729] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a20fac9dff44e2a7d71ff3a22939c7, State: Initialized, Role: FOLLOWER
I20260620 18:42:23.471125 27627 consensus_queue.cc:260] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } }
I20260620 18:42:23.471448 27627 ts_tablet_manager.cc:1428] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:42:23.471482 27618 heartbeater.cc:503] Master 127.10.250.126:44805 was elected leader, sending a full tablet report...
W20260620 18:42:23.533361 27483 tablet.cc:2369] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:42:23.596320 27349 raft_consensus.cc:482] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:23.596499 27349 raft_consensus.cc:504] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } }
I20260620 18:42:23.597018 27349 leader_election.cc:288] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67:36547), 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653)
I20260620 18:42:23.600124 27345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.597284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:36547 (local address 127.10.250.65:56919)
0620 18:42:23.597359 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:23.597374 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:23.597463 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:23.597620 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:23.597621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:23.597622 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:23.597676 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.597678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.598368 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.598369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.599273 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.599276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.599821 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.599822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.599897 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.599931 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:42:23.600010 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 18:42:23.600051 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 18:42:23.600124 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.597413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:56919 (local address 127.10.250.67:36547)
0620 18:42:23.597467 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.597470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.597479 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:42:23.597539 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.597540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.597550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.597605 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:23.597763 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.598276 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.599359 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.599726 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.599823 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.599832 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.600017 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.600056 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 18:42:23.600378 27488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.597394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:48509 (local address 127.10.250.66:46653)
0620 18:42:23.597450 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.597452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.597654 (+ 202us) server_negotiation.cc:407] Connection header received
0620 18:42:23.597783 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.597784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.597815 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.597892 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:23.598078 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.598637 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.599576 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.599979 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.600080 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.600173 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.600240 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.600300 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 18:42:23.600533 27572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ef85a3f1f4452bb0cb0d016c6d78c9" candidate_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" is_pre_election: true
I20260620 18:42:23.600641 27630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.597284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:46653 (local address 127.10.250.65:48509)
0620 18:42:23.597562 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:23.597570 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:23.597689 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:23.597893 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:23.597894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:23.597896 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:23.597996 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.597998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.598726 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.598728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.599471 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.599473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.600078 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.600079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.600148 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.600151 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:42:23.600506 (+ 355us) client_negotiation.cc:240] Negotiation successful
0620 18:42:23.600550 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":88,"thread_start_us":103,"threads_started":1}
I20260620 18:42:23.600720 27572 raft_consensus.cc:2454] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09e1ea8f3b2243e1be7127c62a05ed49 in term 0.
I20260620 18:42:23.600953 27232 leader_election.cc:302] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09e1ea8f3b2243e1be7127c62a05ed49, 91a20fac9dff44e2a7d71ff3a22939c7; no voters:
I20260620 18:42:23.600940 27436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ef85a3f1f4452bb0cb0d016c6d78c9" candidate_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" is_pre_election: true
I20260620 18:42:23.601109 27436 raft_consensus.cc:2454] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09e1ea8f3b2243e1be7127c62a05ed49 in term 0.
I20260620 18:42:23.601097 27349 raft_consensus.cc:2781] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:42:23.601174 27349 raft_consensus.cc:482] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:42:23.601204 27349 raft_consensus.cc:3037] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:23.601970 27349 raft_consensus.cc:504] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } }
I20260620 18:42:23.602139 27349 leader_election.cc:288] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 1 election: Requested vote from peers 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67:36547), 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653)
I20260620 18:42:23.602317 27436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ef85a3f1f4452bb0cb0d016c6d78c9" candidate_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb"
I20260620 18:42:23.602276 27572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ef85a3f1f4452bb0cb0d016c6d78c9" candidate_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91a20fac9dff44e2a7d71ff3a22939c7"
I20260620 18:42:23.602420 27436 raft_consensus.cc:3037] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:23.602447 27572 raft_consensus.cc:3037] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:23.603168 27436 raft_consensus.cc:2454] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09e1ea8f3b2243e1be7127c62a05ed49 in term 1.
I20260620 18:42:23.603230 27572 raft_consensus.cc:2454] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09e1ea8f3b2243e1be7127c62a05ed49 in term 1.
I20260620 18:42:23.603435 27232 leader_election.cc:302] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09e1ea8f3b2243e1be7127c62a05ed49, 91a20fac9dff44e2a7d71ff3a22939c7; no voters:
I20260620 18:42:23.603536 27349 raft_consensus.cc:2781] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:23.603715 27349 raft_consensus.cc:686] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 LEADER]: Becoming Leader. State: Replica: 09e1ea8f3b2243e1be7127c62a05ed49, State: Running, Role: LEADER
I20260620 18:42:23.603780 27349 consensus_queue.cc:237] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } }
I20260620 18:42:23.604571 27155 catalog_manager.cc:5495] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09e1ea8f3b2243e1be7127c62a05ed49 (127.10.250.65). New cstate: current_term: 1 leader_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } health_report { overall_health: HEALTHY } } }
W20260620 18:42:23.642989 27344 tablet.cc:2369] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:42:23.643218 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.640604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45990 (local address 127.10.250.65:36959)
0620 18:42:23.640682 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.640684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.640692 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:42:23.640730 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.640730 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.640748 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.640821 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:42:23.641024 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.641559 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.642334 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.642579 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.642634 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.642816 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:42:23.642921 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:42:23.642977 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.643110 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.643137 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:42:23.644755 27572 raft_consensus.cc:1261] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 1 FOLLOWER]: Refusing update from remote peer 09e1ea8f3b2243e1be7127c62a05ed49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:23.644757 27436 raft_consensus.cc:1261] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 1 FOLLOWER]: Refusing update from remote peer 09e1ea8f3b2243e1be7127c62a05ed49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:23.645146 27631 consensus_queue.cc:1035] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:23.645280 27349 consensus_queue.cc:1035] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:23.647267 27634 mvcc.cc:205] Tried to move back new op lower bound from 7298993945166286848 to 7298993945001795584. Current Snapshot: MvccSnapshot[applied={T|T < 7298993945166286848}]
I20260620 18:42:23.647599 27636 mvcc.cc:205] Tried to move back new op lower bound from 7298993945166286848 to 7298993945001795584. Current Snapshot: MvccSnapshot[applied={T|T < 7298993945166286848}]
I20260620 18:42:23.649549 27155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:23.649642 27155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:23.650329 27155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Sending DeleteTablet for 3 replicas of tablet 01ef85a3f1f4452bb0cb0d016c6d78c9
I20260620 18:42:23.650691 27416 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ef85a3f1f4452bb0cb0d016c6d78c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:23 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:49642
I20260620 18:42:23.650789 27489 tablet_replica.cc:330] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: stopping tablet replica
I20260620 18:42:23.650769 27277 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ef85a3f1f4452bb0cb0d016c6d78c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:23 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:45982
I20260620 18:42:23.650750 27552 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ef85a3f1f4452bb0cb0d016c6d78c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:23 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:56636
I20260620 18:42:23.650837 27489 raft_consensus.cc:2229] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:23.650842 27352 tablet_replica.cc:330] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49: stopping tablet replica
I20260620 18:42:23.650882 27489 raft_consensus.cc:2258] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:23.650882 27352 raft_consensus.cc:2229] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:42:23.650970 27352 raft_consensus.cc:2258] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:23.651269 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 27193
I20260620 18:42:23.651445 27489 ts_tablet_manager.cc:1905] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:23.653192 27489 ts_tablet_manager.cc:1918] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:23.653280 27489 log.cc:1196] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/01ef85a3f1f4452bb0cb0d016c6d78c9
I20260620 18:42:23.653509 27489 ts_tablet_manager.cc:1939] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 1e942e3f2c6c4691aa2b67b9b3e4dcfb: Deleting consensus metadata
I20260620 18:42:23.653878 27140 catalog_manager.cc:4841] TS 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653): tablet 01ef85a3f1f4452bb0cb0d016c6d78c9 (table test-table [id=a17d0dec48ce4ac59354e5d880efc787]) successfully deleted
W20260620 18:42:23.657895 27142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:36959 (error 108)
I20260620 18:42:23.658106 27643 tablet_replica.cc:330] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7: stopping tablet replica
I20260620 18:42:23.658257 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 27347
I20260620 18:42:23.658270 27643 raft_consensus.cc:2229] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:23.658411 27643 raft_consensus.cc:2258] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:23.658941 27350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.658345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:36959 (local address 127.0.0.1:45996)
0620 18:42:23.658842 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:23.658902 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470}
W20260620 18:42:23.659085 27142 catalog_manager.cc:4584] TS 09e1ea8f3b2243e1be7127c62a05ed49 (127.10.250.65:36959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01ef85a3f1f4452bb0cb0d016c6d78c9: Network error: Client connection negotiation failed: client connection to 127.10.250.65:36959: connect: Connection refused (error 111)
I20260620 18:42:23.659825 27643 ts_tablet_manager.cc:1905] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:23.661342 27643 ts_tablet_manager.cc:1918] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:23.661536 27643 log.cc:1196] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/01ef85a3f1f4452bb0cb0d016c6d78c9
I20260620 18:42:23.661798 27643 ts_tablet_manager.cc:1939] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 91a20fac9dff44e2a7d71ff3a22939c7: Deleting consensus metadata
I20260620 18:42:23.662287 27142 catalog_manager.cc:4841] TS 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67:36547): tablet 01ef85a3f1f4452bb0cb0d016c6d78c9 (table test-table [id=a17d0dec48ce4ac59354e5d880efc787]) successfully deleted
I20260620 18:42:23.664012 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 27485
I20260620 18:42:23.668998 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 27122
I20260620 18:42:43.674697 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44805
--webserver_interface=127.10.250.126
--webserver_port=44125
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:44805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:43.757855 27645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:43.758088 27645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:43.758112 27645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:43.759290 27645 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:42:43.759332 27645 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:42:43.759346 27645 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:42:43.759357 27645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:43.759402 27645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:43.759426 27645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:43.759444 27645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:43.759476 27645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:43.760987 27645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:44805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=44125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:42:43.761200 27645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:43.761384 27645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:43.766551 27651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:43.766644 27645 server_base.cc:1034] running on GCE node
W20260620 18:42:43.766747 27650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:43.766584 27653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:43.767055 27645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:43.767369 27645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:43.768532 27645 hybrid_clock.cc:648] HybridClock initialized: now 1781980963768499 us; error 46 us; skew 500 ppm
Jun 20 18:42:43 dist-test-slave-wp6g krb5kdc[26575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980963, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:43.772231 27645 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:42:43.772828 27645 webserver.cc:460] Webserver started at http://127.10.250.126:44125/ using document root <none> and password file <none>
I20260620 18:42:43.773100 27645 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:43.773185 27645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:43.774708 27645 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:42:43.775475 27660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:43.775647 27645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:42:43.775741 27645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "cae28bf3223a4903a5ee281684216b1f"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "88be9f89db06ad092804d321d8782e68"
server_key_iv: "1ae82d7d2a8c3007e1e9ca2f8ab02d24"
server_key_version: "encryptionkey@0"
I20260620 18:42:43.776100 27645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:43.796582 27645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:43.798707 27645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:43.798826 27645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:43.803109 27645 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:44805
I20260620 18:42:43.803210 27712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:44805 every 8 connection(s)
I20260620 18:42:43.803593 27645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:43.805176 27713 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:42:43.805841 27713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Bootstrap starting.
I20260620 18:42:43.810295 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 27645
I20260620 18:42:43.810691 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:36959
--local_ip_for_outbound_sockets=127.10.250.65
--tserver_master_addrs=127.10.250.126:44805
--webserver_port=40779
--webserver_interface=127.10.250.65
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:42:43.814939 27713 log.cc:824] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:43.817381 27713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:42:43.817749 27713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Bootstrap complete.
I20260620 18:42:43.820076 27713 raft_consensus.cc:348] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } }
I20260620 18:42:43.820333 27713 raft_consensus.cc:729] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae28bf3223a4903a5ee281684216b1f, State: Initialized, Role: FOLLOWER
I20260620 18:42:43.820533 27713 consensus_queue.cc:260] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } }
I20260620 18:42:43.820626 27713 raft_consensus.cc:388] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:43.820678 27713 raft_consensus.cc:482] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:43.820726 27713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:42:43.821758 27713 raft_consensus.cc:504] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } }
I20260620 18:42:43.821877 27713 leader_election.cc:302] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cae28bf3223a4903a5ee281684216b1f; no voters:
I20260620 18:42:43.822081 27713 leader_election.cc:288] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 18:42:43.822119 27717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:42:43.822229 27717 raft_consensus.cc:686] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [term 3 LEADER]: Becoming Leader. State: Replica: cae28bf3223a4903a5ee281684216b1f, State: Running, Role: LEADER
I20260620 18:42:43.822316 27717 consensus_queue.cc:237] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } }
I20260620 18:42:43.822358 27713 sys_catalog.cc:564] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:43.822638 27717 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae28bf3223a4903a5ee281684216b1f. Latest consensus state: current_term: 3 leader_uuid: "cae28bf3223a4903a5ee281684216b1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } } }
I20260620 18:42:43.822710 27717 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:43.822858 27718 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cae28bf3223a4903a5ee281684216b1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae28bf3223a4903a5ee281684216b1f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44805 } } }
I20260620 18:42:43.822943 27718 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:43.823213 27721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:43.824092 27721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:43.824229 27721 catalog_manager.cc:1259] Loaded cluster ID: 9c157558f6464ebebc10107d39aeea0b
I20260620 18:42:43.824273 27721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:43.825245 27721 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:43.825438 27721 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Loaded TSK: 0
I20260620 18:42:43.825492 27721 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cae28bf3223a4903a5ee281684216b1f: Loaded TSK: 1
F20260620 18:42:43.825711 27721 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6b84d229ad google::LogMessage::Fail() at ??:0
@ 0x7f6b84d2588a google::LogMessage::SendToLog() at ??:0
@ 0x7f6b84d224de google::LogMessage::Flush() at ??:0
@ 0x7f6b84d23a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6b8748f104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6b8748cd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f6b8cc7c1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f6b8cc7c48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f6b8cc7c594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f6b8cc42c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f6b8cc4cad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6b86bf00d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6b86be5954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6b87a496db start_thread at ??:0
@ 0x7f6b8382061f clone at ??:0
W20260620 18:42:43.916777 27715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:43.916991 27715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:43.917016 27715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:43.918255 27715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:43.918301 27715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:43.918329 27715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:42:43.919873 27715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:36959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=40779
--tserver_master_addrs=127.10.250.126:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:42:43.920137 27715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:43.920380 27715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:43.926723 27755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:43.926703 27754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:43.926708 27757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:43.927022 27715 server_base.cc:1034] running on GCE node
I20260620 18:42:43.927294 27715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:43.927590 27715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:43.928763 27715 hybrid_clock.cc:648] HybridClock initialized: now 1781980963928750 us; error 31 us; skew 500 ppm
Jun 20 18:42:43 dist-test-slave-wp6g krb5kdc[26575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980963, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:43.931574 27715 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:42:43.932116 27715 webserver.cc:460] Webserver started at http://127.10.250.65:40779/ using document root <none> and password file <none>
I20260620 18:42:43.932372 27715 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:43.932441 27715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:43.933892 27715 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:42:43.936208 27764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:43.936475 27715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:42:43.936599 27715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "09e1ea8f3b2243e1be7127c62a05ed49"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "60f545429e3fa66fe6ddd29076d927ad"
server_key_iv: "60d9c72acc85d964fc9353576035cdb1"
server_key_version: "encryptionkey@0"
I20260620 18:42:43.937036 27715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:43.950982 27715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:43.953483 27715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:43.953655 27715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:43.953905 27715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:43.954459 27771 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 18:42:43.956604 27715 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 18:42:43.956707 27715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 18:42:43.956759 27715 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 18:42:43.957625 27715 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 18:42:43.957692 27715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 18:42:43.957767 27771 tablet_bootstrap.cc:492] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49: Bootstrap starting.
I20260620 18:42:43.965972 27715 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:36959
I20260620 18:42:43.966120 27878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:36959 every 8 connection(s)
I20260620 18:42:43.966459 27715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:42:43.969309 27771 log.cc:824] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:43.972368 27771 tablet_bootstrap.cc:492] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:42:43.972747 27771 tablet_bootstrap.cc:492] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49: Bootstrap complete.
I20260620 18:42:43.973992 27771 ts_tablet_manager.cc:1397] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260620 18:42:43.975561 27771 raft_consensus.cc:348] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } }
I20260620 18:42:43.975920 27771 raft_consensus.cc:729] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e1ea8f3b2243e1be7127c62a05ed49, State: Initialized, Role: FOLLOWER
I20260620 18:42:43.976080 27771 consensus_queue.cc:260] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } }
I20260620 18:42:43.976400 27771 ts_tablet_manager.cc:1428] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
I20260620 18:42:43.976616 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 27715
I20260620 18:42:43.976895 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:46653
--local_ip_for_outbound_sockets=127.10.250.66
--tserver_master_addrs=127.10.250.126:44805
--webserver_port=41165
--webserver_interface=127.10.250.66
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:44.031975 27881 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:43.967386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:46903)
0620 18:42:43.967604 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:43.967686 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:43.968207 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:44.031751 (+ 63544us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":124,"threads_started":1}
W20260620 18:42:44.032207 27879 heartbeater.cc:644] Failed to heartbeat to 127.10.250.126:44805 (0 consecutive failures): Network error: Failed to ping master at 127.10.250.126:44805: Client connection negotiation failed: client connection to 127.10.250.126:44805: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 18:42:44.032543 27881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:44.032417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:59673)
0620 18:42:44.032470 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:44.032518 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 18:42:44.032944 27881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:44.032861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:59689)
0620 18:42:44.032906 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:44.032929 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 18:42:44.033108 27879 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:42:44.067294 27884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:44.067545 27884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:44.067588 27884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:44.068799 27884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:44.068863 27884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:44.068914 27884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:42:44.070441 27884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:46653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=41165
--tserver_master_addrs=127.10.250.126:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:42:44.070747 27884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:44.070981 27884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:44.076243 27889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:44.076251 27892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:44.076280 27884 server_base.cc:1034] running on GCE node
W20260620 18:42:44.076292 27890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:44.076706 27884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:44.076987 27884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:44.078163 27884 hybrid_clock.cc:648] HybridClock initialized: now 1781980964078147 us; error 38 us; skew 500 ppm
Jun 20 18:42:44 dist-test-slave-wp6g krb5kdc[26575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980964, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:44.080756 27884 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:42:44.081195 27884 webserver.cc:460] Webserver started at http://127.10.250.66:41165/ using document root <none> and password file <none>
I20260620 18:42:44.081396 27884 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:44.081451 27884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:44.082882 27884 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:42:44.083546 27899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:44.083737 27884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:42:44.083843 27884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "f135ed6d060eed10a1cbc13c449594ba"
server_key_iv: "5bd14d5dac5d803ef139d15ed9354416"
server_key_version: "encryptionkey@0"
I20260620 18:42:44.084231 27884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:44.105710 27884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:44.108320 27884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:44.108506 27884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:44.108757 27884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:44.109081 27884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:44.109134 27884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:44.109177 27884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:44.109203 27884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:44.115267 27884 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:46653
I20260620 18:42:44.115382 28012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:46653 every 8 connection(s)
I20260620 18:42:44.115710 27884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:42:44.117120 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:44.116458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:39461)
0620 18:42:44.116615 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:44.116694 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
W20260620 18:42:44.117374 28013 heartbeater.cc:644] Failed to heartbeat to 127.10.250.126:44805 (0 consecutive failures): Network error: Failed to ping master at 127.10.250.126:44805: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
I20260620 18:42:44.117650 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:44.117561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:35937)
0620 18:42:44.117603 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:44.117631 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 18:42:44.117901 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:44.117831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:46985)
0620 18:42:44.117866 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:44.117888 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 18:42:44.118048 28013 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:42:44.122917 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 27884
I20260620 18:42:44.123154 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:36547
--local_ip_for_outbound_sockets=127.10.250.67
--tserver_master_addrs=127.10.250.126:44805
--webserver_port=41079
--webserver_interface=127.10.250.67
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:44.206079 28016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:44.206332 28016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:44.206362 28016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:44.207813 28016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:44.207873 28016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:44.207913 28016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:42:44.209628 28016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:36547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=41079
--tserver_master_addrs=127.10.250.126:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:42:44.209885 28016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:44.210162 28016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:44.216063 28021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:44.216075 28022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:44.215998 28024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:44.216306 28016 server_base.cc:1034] running on GCE node
I20260620 18:42:44.216506 28016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:44.216753 28016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:44.217926 28016 hybrid_clock.cc:648] HybridClock initialized: now 1781980964217908 us; error 29 us; skew 500 ppm
Jun 20 18:42:44 dist-test-slave-wp6g krb5kdc[26575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980964, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:44.220700 28016 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:42:44.221194 28016 webserver.cc:460] Webserver started at http://127.10.250.67:41079/ using document root <none> and password file <none>
I20260620 18:42:44.221438 28016 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:44.221493 28016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:44.223003 28016 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:42:44.223743 28031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:44.223949 28016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:42:44.224093 28016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "91a20fac9dff44e2a7d71ff3a22939c7"
format_stamp: "Formatted at 2026-06-20 18:42:02 on dist-test-slave-wp6g"
server_key: "6c4307d473c2316c3f431c47996656b7"
server_key_iv: "66d70f9b23f261b36e3223fa13b825bc"
server_key_version: "encryptionkey@0"
I20260620 18:42:44.224516 28016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:44.244583 28016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:44.247054 28016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:44.247232 28016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:44.247499 28016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:44.247828 28016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:44.247880 28016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:44.247921 28016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:44.247958 28016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:44.254624 28016 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:36547
I20260620 18:42:44.254683 28144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:36547 every 8 connection(s)
I20260620 18:42:44.255079 28016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:42:44.256203 28147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:44.255800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:43805)
0620 18:42:44.255951 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:44.256024 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
W20260620 18:42:44.256628 28145 heartbeater.cc:644] Failed to heartbeat to 127.10.250.126:44805 (0 consecutive failures): Network error: Failed to ping master at 127.10.250.126:44805: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
I20260620 18:42:44.256894 28147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:44.256809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:58493)
0620 18:42:44.256850 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:44.256877 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 18:42:44.257201 28147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:44.257125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:57945)
0620 18:42:44.257164 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:44.257187 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 18:42:44.257318 28145 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:42:44.258298 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 28016
I20260620 18:42:45.033818 28149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:45.033524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:41857)
0620 18:42:45.033699 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:45.033754 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20260620 18:42:45.118656 28150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:45.118395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:58131)
0620 18:42:45.118569 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:45.118610 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260620 18:42:45.154713 28151 raft_consensus.cc:482] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:45.154873 28151 raft_consensus.cc:504] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } }
I20260620 18:42:45.155305 28151 leader_election.cc:288] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67:36547), 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653)
Jun 20 18:42:45 dist-test-slave-wp6g krb5kdc[26575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980963, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.67@KRBTEST.COM
Jun 20 18:42:45 dist-test-slave-wp6g krb5kdc[26575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980963, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.66@KRBTEST.COM
I20260620 18:42:45.166155 28153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:45.155603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:57877 (local address 127.10.250.67:36547)
0620 18:42:45.155776 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:45.155788 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:45.155808 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:42:45.156279 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:45.156281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:45.156338 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:45.156498 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:45.157253 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:45.158167 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:45.159320 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:45.159736 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:45.159870 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:45.162399 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:45.162403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:45.162403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:45.162415 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:45.164382 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:45.164894 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:45.164897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:45.164898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:45.164995 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:45.165272 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:45.165273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:45.165273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:45.165513 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:45.165608 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:45.165901 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 18:42:45.166040 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260620 18:42:45.166155 28149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:45.155493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:36547 (local address 127.10.250.65:57877)
0620 18:42:45.155552 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:45.155564 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:45.155624 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:45.156574 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:45.156577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:45.156948 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:45.157123 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:45.157128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:45.158355 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:45.158356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:45.159227 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:45.159230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:45.159841 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:45.159842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:45.159970 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:45.160598 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:45.160607 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:45.162261 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:45.164490 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:45.164498 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:45.164500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:45.164800 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:45.165080 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:45.165081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:45.165081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:45.165196 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:45.165587 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:45.165594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:45.165743 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 18:42:45.165851 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 18:42:45.166053 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":43}
I20260620 18:42:45.166173 28154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:45.155650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:40019 (local address 127.10.250.66:46653)
0620 18:42:45.155788 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:45.155803 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:45.155832 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:42:45.156289 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:45.156292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:45.156336 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:45.156496 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:45.157274 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:45.158155 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:45.159320 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:45.159736 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:45.159870 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:45.162871 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:45.162876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:45.162876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:45.162886 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:45.164490 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:45.164915 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:45.164918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:45.164919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:45.165023 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:45.165260 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:45.165261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:45.165262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:45.165515 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:45.165608 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:45.165899 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 18:42:45.166039 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":53,"threads_started":1}
I20260620 18:42:45.166352 28152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:45.155497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:46653 (local address 127.10.250.65:40019)
0620 18:42:45.155640 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:45.155651 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:45.155693 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:45.156606 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:45.156607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:45.156923 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:45.157123 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:45.157129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:45.158486 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:45.158487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:45.159227 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:45.159229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:45.159846 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:45.159847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:45.159977 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:45.160635 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:45.160639 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:45.162741 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:45.164603 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:45.164604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:45.164604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:45.164828 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:45.165101 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:45.165102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:45.165102 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:45.165197 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:45.165590 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:45.165596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:45.165744 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 18:42:45.165851 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 18:42:45.166135 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":126,"thread_start_us":58,"threads_started":1}
W20260620 18:42:45.167222 27768 leader_election.cc:341] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67:36547): Not found: Tablet not found: 01ef85a3f1f4452bb0cb0d016c6d78c9
W20260620 18:42:45.167311 27766 leader_election.cc:341] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653): Not found: Tablet not found: 01ef85a3f1f4452bb0cb0d016c6d78c9
I20260620 18:42:45.167373 27766 leader_election.cc:302] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09e1ea8f3b2243e1be7127c62a05ed49; no voters: 1e942e3f2c6c4691aa2b67b9b3e4dcfb, 91a20fac9dff44e2a7d71ff3a22939c7
I20260620 18:42:45.167482 28151 raft_consensus.cc:2726] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:42:45.257937 28155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:45.257692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:55095)
0620 18:42:45.257832 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:45.257897 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260620 18:42:46.034749 28156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:46.034425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:46039)
0620 18:42:46.034653 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:46.034700 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":74,"threads_started":1}
I20260620 18:42:46.119379 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:46.119140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:39041)
0620 18:42:46.119295 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:46.119335 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20260620 18:42:46.258742 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:46.258459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:32951)
0620 18:42:46.258656 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:46.258697 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":133,"threads_started":1}
I20260620 18:42:47.035526 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:47.035275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:47183)
0620 18:42:47.035433 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:47.035479 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
I20260620 18:42:47.067094 28160 raft_consensus.cc:482] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:47.067221 28160 raft_consensus.cc:504] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } }
I20260620 18:42:47.067572 28160 leader_election.cc:288] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67:36547), 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653)
W20260620 18:42:47.067910 27766 leader_election.cc:341] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653): Not found: Tablet not found: 01ef85a3f1f4452bb0cb0d016c6d78c9
W20260620 18:42:47.067992 27768 leader_election.cc:341] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67:36547): Not found: Tablet not found: 01ef85a3f1f4452bb0cb0d016c6d78c9
I20260620 18:42:47.068077 27766 leader_election.cc:302] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09e1ea8f3b2243e1be7127c62a05ed49; no voters: 1e942e3f2c6c4691aa2b67b9b3e4dcfb, 91a20fac9dff44e2a7d71ff3a22939c7
I20260620 18:42:47.068177 28160 raft_consensus.cc:2726] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:42:47.120112 28161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:47.119870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:34011)
0620 18:42:47.120008 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:47.120067 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260620 18:42:47.259415 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:47.259217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:42865)
0620 18:42:47.259335 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:47.259376 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260620 18:42:48.036320 28163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:48.036066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:35777)
0620 18:42:48.036219 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:48.036273 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260620 18:42:48.120968 28164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:48.120729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:45119)
0620 18:42:48.120875 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:48.120921 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260620 18:42:48.260223 28165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:48.259939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:44385)
0620 18:42:48.260132 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:48.260178 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":110,"threads_started":1}
I20260620 18:42:49.037147 28166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:49.036868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:39189)
0620 18:42:49.037048 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:49.037094 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
I20260620 18:42:49.122004 28167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:49.121678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:49449)
0620 18:42:49.121876 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:49.121936 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":62,"threads_started":1}
I20260620 18:42:49.261020 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:49.260779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:44215)
0620 18:42:49.260930 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:49.260966 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260620 18:42:49.690593 28169 raft_consensus.cc:482] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:49.690714 28169 raft_consensus.cc:504] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } }
I20260620 18:42:49.691005 28169 leader_election.cc:288] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67:36547), 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653)
W20260620 18:42:49.691546 27766 leader_election.cc:341] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653): Not found: Tablet not found: 01ef85a3f1f4452bb0cb0d016c6d78c9
W20260620 18:42:49.691637 27768 leader_election.cc:341] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67:36547): Not found: Tablet not found: 01ef85a3f1f4452bb0cb0d016c6d78c9
I20260620 18:42:49.691672 27768 leader_election.cc:302] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09e1ea8f3b2243e1be7127c62a05ed49; no voters: 1e942e3f2c6c4691aa2b67b9b3e4dcfb, 91a20fac9dff44e2a7d71ff3a22939c7
I20260620 18:42:49.691787 28169 raft_consensus.cc:2726] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:42:50.038036 28170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:50.037746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:34675)
0620 18:42:50.037931 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:50.037987 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":87,"threads_started":1}
I20260620 18:42:50.123049 28171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:50.122764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:50033)
0620 18:42:50.122917 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:50.122966 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260620 18:42:50.261973 28172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:50.261731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:34447)
0620 18:42:50.261879 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:50.261927 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260620 18:42:51.038975 28173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:51.038722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:53553)
0620 18:42:51.038873 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:51.038925 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
I20260620 18:42:51.123901 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:51.123656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:57383)
0620 18:42:51.123808 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:51.123853 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260620 18:42:51.262962 28175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:51.262654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:39531)
0620 18:42:51.262861 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:51.262916 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260620 18:42:52.039816 28176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:52.039554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:54667)
0620 18:42:52.039720 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:52.039768 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260620 18:42:52.124637 28177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:52.124428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:45135)
0620 18:42:52.124551 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:52.124593 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260620 18:42:52.232573 28178 raft_consensus.cc:482] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:52.232685 28178 raft_consensus.cc:504] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } }
I20260620 18:42:52.232926 28178 leader_election.cc:288] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67:36547), 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653)
W20260620 18:42:52.233357 27768 leader_election.cc:341] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67:36547): Not found: Tablet not found: 01ef85a3f1f4452bb0cb0d016c6d78c9
W20260620 18:42:52.233424 27766 leader_election.cc:341] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653): Not found: Tablet not found: 01ef85a3f1f4452bb0cb0d016c6d78c9
I20260620 18:42:52.233458 27768 leader_election.cc:302] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09e1ea8f3b2243e1be7127c62a05ed49; no voters: 1e942e3f2c6c4691aa2b67b9b3e4dcfb, 91a20fac9dff44e2a7d71ff3a22939c7
I20260620 18:42:52.233556 28178 raft_consensus.cc:2726] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:42:52.263782 28179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:52.263509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:58423)
0620 18:42:52.263673 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:52.263722 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
I20260620 18:42:53.040506 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:53.040298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:38897)
0620 18:42:53.040430 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:53.040461 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260620 18:42:53.125371 28181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:53.125131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:50237)
0620 18:42:53.125271 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:53.125325 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260620 18:42:53.264529 28182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:53.264297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:35497)
0620 18:42:53.264435 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:53.264483 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260620 18:42:54.041440 28183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:54.041175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:48491)
0620 18:42:54.041342 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:54.041392 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":67,"threads_started":1}
I20260620 18:42:54.126211 28184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:54.125956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:42073)
0620 18:42:54.126120 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:54.126164 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
I20260620 18:42:54.265422 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:54.265143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:53725)
0620 18:42:54.265309 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:54.265364 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20260620 18:42:55.042341 28186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:55.042048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:37365)
0620 18:42:55.042244 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:55.042293 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260620 18:42:55.127123 28187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:55.126815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:40465)
0620 18:42:55.127029 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:55.127077 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":120,"threads_started":1}
I20260620 18:42:55.254618 28188 raft_consensus.cc:482] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:55.254779 28188 raft_consensus.cc:504] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a20fac9dff44e2a7d71ff3a22939c7" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36547 } } peers { permanent_uuid: "1e942e3f2c6c4691aa2b67b9b3e4dcfb" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 46653 } } peers { permanent_uuid: "09e1ea8f3b2243e1be7127c62a05ed49" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36959 } }
I20260620 18:42:55.254972 28188 leader_election.cc:288] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67:36547), 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653)
W20260620 18:42:55.255342 27768 leader_election.cc:341] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91a20fac9dff44e2a7d71ff3a22939c7 (127.10.250.67:36547): Not found: Tablet not found: 01ef85a3f1f4452bb0cb0d016c6d78c9
W20260620 18:42:55.255450 27766 leader_election.cc:341] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e942e3f2c6c4691aa2b67b9b3e4dcfb (127.10.250.66:46653): Not found: Tablet not found: 01ef85a3f1f4452bb0cb0d016c6d78c9
I20260620 18:42:55.255548 27766 leader_election.cc:302] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09e1ea8f3b2243e1be7127c62a05ed49; no voters: 1e942e3f2c6c4691aa2b67b9b3e4dcfb, 91a20fac9dff44e2a7d71ff3a22939c7
I20260620 18:42:55.255720 28188 raft_consensus.cc:2726] T 01ef85a3f1f4452bb0cb0d016c6d78c9 P 09e1ea8f3b2243e1be7127c62a05ed49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:42:55.266342 28189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:55.266077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:53973)
0620 18:42:55.266243 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:55.266291 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260620 18:42:56.043107 28190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:56.042863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:39559)
0620 18:42:56.043009 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:56.043058 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260620 18:42:56.128069 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:56.127715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:59153)
0620 18:42:56.127910 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:56.127952 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":116,"threads_started":1}
I20260620 18:42:56.267153 28192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:56.266892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:36839)
0620 18:42:56.267055 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:56.267105 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260620 18:42:57.044037 28193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:57.043709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:60247)
0620 18:42:57.043948 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:57.043989 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
I20260620 18:42:57.128885 28194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:57.128637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:41693)
0620 18:42:57.128796 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:57.128840 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
I20260620 18:42:57.268096 28195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:57.267833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:34965)
0620 18:42:57.267983 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:57.268044 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260620 18:42:58.044703 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:58.044478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:33643)
0620 18:42:58.044617 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:58.044658 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260620 18:42:58.129593 28197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:58.129338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:55581)
0620 18:42:58.129505 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:58.129545 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
I20260620 18:42:58.269150 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:58.268885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:32853)
0620 18:42:58.269055 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:58.269103 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
I20260620 18:42:59.045531 28199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.045302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.65:54733)
0620 18:42:59.045446 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:59.045486 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260620 18:42:59.130416 28200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.130191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.66:48365)
0620 18:42:59.130329 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:59.130371 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260620 18:42:59.269100 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 27715
I20260620 18:42:59.270715 28201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.270463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44805 (local address 127.10.250.67:60311)
0620 18:42:59.270608 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:59.270644 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.126:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260620 18:42:59.274210 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 27884
I20260620 18:42:59.278750 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 28016
2026-06-20T18:42:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58499 ms)
[----------] 30 tests from SecurityITest (240598 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:42:59 dist-test-slave-wp6g krb5kdc[28204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:42:59 dist-test-slave-wp6g krb5kdc[28204](info): set up 2 sockets
Jun 20 18:42:59 dist-test-slave-wp6g krb5kdc[28204](info): commencing operation
krb5kdc: starting...
W20260620 18:43:00.354373 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:00 dist-test-slave-wp6g krb5kdc[28204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:43:00.482373 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38837
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:38837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:00.567165 28217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:00.567467 28217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:00.567516 28217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:00.568858 28217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:00.568940 28217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:00.568966 28217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:00.568989 28217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:00.569018 28217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:00.570727 28217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:38837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:00.571054 28217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:00.571309 28217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:43:00.577412 28217 server_base.cc:1034] running on GCE node
W20260620 18:43:00.577414 28225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:00.577414 28223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:00.577538 28222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:00.578398 28217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:00.578799 28217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:00.580021 28217 hybrid_clock.cc:648] HybridClock initialized: now 1781980980579997 us; error 48 us; skew 500 ppm
Jun 20 18:43:00 dist-test-slave-wp6g krb5kdc[28204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980980, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:00.583280 28217 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:43:00.583814 28217 webserver.cc:460] Webserver started at http://127.10.250.126:39825/ using document root <none> and password file <none>
I20260620 18:43:00.584077 28217 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:00.584153 28217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:00.584319 28217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:00.585490 28217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "877ff278abcd4d4ebba04f3d865ec59b"
format_stamp: "Formatted at 2026-06-20 18:43:00 on dist-test-slave-wp6g"
server_key: "e015053f4ac02799b554caf1c4d0733a"
server_key_iv: "a969559b1327167bc14937a3a456fb00"
server_key_version: "encryptionkey@0"
I20260620 18:43:00.585880 28217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "877ff278abcd4d4ebba04f3d865ec59b"
format_stamp: "Formatted at 2026-06-20 18:43:00 on dist-test-slave-wp6g"
server_key: "e015053f4ac02799b554caf1c4d0733a"
server_key_iv: "a969559b1327167bc14937a3a456fb00"
server_key_version: "encryptionkey@0"
I20260620 18:43:00.587473 28217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:43:00.588410 28232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:00.588657 28217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:00.588778 28217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "877ff278abcd4d4ebba04f3d865ec59b"
format_stamp: "Formatted at 2026-06-20 18:43:00 on dist-test-slave-wp6g"
server_key: "e015053f4ac02799b554caf1c4d0733a"
server_key_iv: "a969559b1327167bc14937a3a456fb00"
server_key_version: "encryptionkey@0"
I20260620 18:43:00.588860 28217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:00.596112 28217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:00.598708 28217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:00.598887 28217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:00.603050 28217 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:38837
I20260620 18:43:00.603081 28284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:38837 every 8 connection(s)
I20260620 18:43:00.603461 28217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:00.604434 28285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:00.606853 28285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b: Bootstrap starting.
I20260620 18:43:00.607451 28285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:00.607743 28285 log.cc:824] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:00.608065 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 28217
I20260620 18:43:00.608187 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:00.608379 11241 external_mini_cluster.cc:1428] Setting key ca3f2f1560ea0db39f7ee0dbeefa5910
I20260620 18:43:00.608646 28285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b: No bootstrap required, opened a new log
I20260620 18:43:00.610930 28285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "877ff278abcd4d4ebba04f3d865ec59b" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38837 } }
I20260620 18:43:00.611129 28285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:00.611176 28285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 877ff278abcd4d4ebba04f3d865ec59b, State: Initialized, Role: FOLLOWER
I20260620 18:43:00.611326 28285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "877ff278abcd4d4ebba04f3d865ec59b" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38837 } }
I20260620 18:43:00.611424 28285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:00.611462 28285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:00.611511 28285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:00.612217 28285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "877ff278abcd4d4ebba04f3d865ec59b" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38837 } }
I20260620 18:43:00.612334 28285 leader_election.cc:302] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 877ff278abcd4d4ebba04f3d865ec59b; no voters:
I20260620 18:43:00.612490 28285 leader_election.cc:288] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:00.612562 28290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:00.612711 28285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:00.612730 28290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [term 1 LEADER]: Becoming Leader. State: Replica: 877ff278abcd4d4ebba04f3d865ec59b, State: Running, Role: LEADER
I20260620 18:43:00.613862 28290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "877ff278abcd4d4ebba04f3d865ec59b" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38837 } }
I20260620 18:43:00.614418 28291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 877ff278abcd4d4ebba04f3d865ec59b. Latest consensus state: current_term: 1 leader_uuid: "877ff278abcd4d4ebba04f3d865ec59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "877ff278abcd4d4ebba04f3d865ec59b" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38837 } } }
I20260620 18:43:00.614735 28292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "877ff278abcd4d4ebba04f3d865ec59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "877ff278abcd4d4ebba04f3d865ec59b" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38837 } } }
I20260620 18:43:00.615552 28291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:00.615583 28292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b [sys.catalog]: This master's current role is: LEADER
W20260620 18:43:00.615243 28304 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:43:00.615736 28304 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:43:00.615861 28307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:00.616055 28307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 18:43:00 dist-test-slave-wp6g krb5kdc[28204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:00.617530 28307 catalog_manager.cc:1347] Generated new cluster ID: de51e760639e44b6a55f1cdb05891cd9
I20260620 18:43:00.617581 28307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:00.621694 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:00.608985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.10.250.126:38837)
0620 18:43:00.609193 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:00.609203 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:00.609237 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:43:00.610034 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:00.610044 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:00.610112 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:00.610269 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:00.611049 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.612757 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.613780 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.614466 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.614604 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:00.617776 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:00.617780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:00.617780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:00.617792 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:00.619781 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:00.620215 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:00.620218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:00.620218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:00.620328 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:00.620545 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:00.620545 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:00.620546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:00.620833 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:00.620978 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:00.621369 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 18:43:00.621498 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":105,"threads_started":1}
I20260620 18:43:00.627023 28307 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:00.627548 28307 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:00.635691 28307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b: Generated new TSK 0
I20260620 18:43:00.635948 28307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:43:00.665894 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38837
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:00.763708 28311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:00.763955 28311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:00.763998 28311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:00.765189 28311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:00.765257 28311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:00.765306 28311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:43:00.766965 28311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:00.767242 28311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:00.767484 28311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:00.773065 28319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:00.773043 28316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:00.773077 28317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:00.773099 28311 server_base.cc:1034] running on GCE node
I20260620 18:43:00.773537 28311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:00.773815 28311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:00.774991 28311 hybrid_clock.cc:648] HybridClock initialized: now 1781980980774989 us; error 41 us; skew 500 ppm
Jun 20 18:43:00 dist-test-slave-wp6g krb5kdc[28204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980980, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:00.777768 28311 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:43:00.778266 28311 webserver.cc:460] Webserver started at http://127.10.250.65:38513/ using document root <none> and password file <none>
I20260620 18:43:00.778501 28311 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:00.778625 28311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:00.778868 28311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:00.779960 28311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "48f8b0786a9541bfad6707f4701634ff"
format_stamp: "Formatted at 2026-06-20 18:43:00 on dist-test-slave-wp6g"
server_key: "4bf2f49f3173456fa36977d0736a6080"
server_key_iv: "01de9547490a12ad52b9f5aff32aad09"
server_key_version: "encryptionkey@0"
I20260620 18:43:00.780328 28311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "48f8b0786a9541bfad6707f4701634ff"
format_stamp: "Formatted at 2026-06-20 18:43:00 on dist-test-slave-wp6g"
server_key: "4bf2f49f3173456fa36977d0736a6080"
server_key_iv: "01de9547490a12ad52b9f5aff32aad09"
server_key_version: "encryptionkey@0"
I20260620 18:43:00.781656 28311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:43:00.782469 28326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:00.782706 28311 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:00.782842 28311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "48f8b0786a9541bfad6707f4701634ff"
format_stamp: "Formatted at 2026-06-20 18:43:00 on dist-test-slave-wp6g"
server_key: "4bf2f49f3173456fa36977d0736a6080"
server_key_iv: "01de9547490a12ad52b9f5aff32aad09"
server_key_version: "encryptionkey@0"
I20260620 18:43:00.782923 28311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:00.794154 28311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:00.796352 28311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:00.796532 28311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:00.796795 28311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:00.797123 28311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:00.797178 28311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:00.797221 28311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:00.797252 28311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:00.803635 28311 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:43827
I20260620 18:43:00.803675 28439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:43827 every 8 connection(s)
I20260620 18:43:00.804046 28311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:43:00 dist-test-slave-wp6g krb5kdc[28204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980980, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:00.812175 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 28311
I20260620 18:43:00.812341 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:43:00.812579 11241 external_mini_cluster.cc:1428] Setting key 61d8deb51b596f4589435dfa59404aaa
I20260620 18:43:00.815135 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:00.804820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:39675 (local address 127.10.250.126:38837)
0620 18:43:00.804875 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:00.804877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:00.805044 (+ 167us) server_negotiation.cc:407] Connection header received
0620 18:43:00.805594 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:00.805596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:00.805608 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:00.805719 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:00.806841 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.807383 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.808512 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.808775 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.808840 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:00.811244 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:00.811245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:00.811246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:00.811254 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:00.813343 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:00.813852 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:00.813853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:00.813853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:00.813926 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:00.814209 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:00.814210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:00.814211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:00.814350 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:00.814452 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:00.814887 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 18:43:00.815005 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:43:00.815251 28442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:00.804736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38837 (local address 127.10.250.65:39675)
0620 18:43:00.804912 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:00.804956 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:00.805476 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:00.805818 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:00.805823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:00.806438 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:00.806736 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:00.806742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.807507 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.807508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:00.808407 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:00.808410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.808895 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.808896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:00.809004 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:00.809534 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:00.809543 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:00.811120 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:00.813423 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:00.813435 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:00.813438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:00.813706 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:00.813995 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:00.813996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:00.813997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:00.814130 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:00.814459 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:00.814467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:00.814703 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 18:43:00.814850 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 18:43:00.815030 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260620 18:43:00.816025 28440 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38837
I20260620 18:43:00.816162 28440 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:00.816372 28440 heartbeater.cc:511] Master 127.10.250.126:38837 requested a full tablet report, sending...
I20260620 18:43:00.817003 28249 ts_manager.cc:194] Registered new tserver with Master: 48f8b0786a9541bfad6707f4701634ff (127.10.250.65:43827)
I20260620 18:43:00.817811 28249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:39675
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:43:00.839769 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38837
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:00.925896 28445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:00.926136 28445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:00.926162 28445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:00.927521 28445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:00.927578 28445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:00.927608 28445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:43:00.929242 28445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:00.929526 28445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:00.929776 28445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:00.935158 28451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:00.935149 28450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:00.935149 28453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:00.935405 28445 server_base.cc:1034] running on GCE node
I20260620 18:43:00.935635 28445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:00.935904 28445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:00.937060 28445 hybrid_clock.cc:648] HybridClock initialized: now 1781980980937043 us; error 35 us; skew 500 ppm
Jun 20 18:43:00 dist-test-slave-wp6g krb5kdc[28204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980980, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:00.939869 28445 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:43:00.940327 28445 webserver.cc:460] Webserver started at http://127.10.250.66:36271/ using document root <none> and password file <none>
I20260620 18:43:00.940567 28445 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:00.940613 28445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:00.940747 28445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:00.941874 28445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "69676d0a402a459f8f63958dde80c256"
format_stamp: "Formatted at 2026-06-20 18:43:00 on dist-test-slave-wp6g"
server_key: "f41870c094b4ad2a687b6e2422282d80"
server_key_iv: "4e07ab1c3bd571f98d38c62e98951c7e"
server_key_version: "encryptionkey@0"
I20260620 18:43:00.942237 28445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "69676d0a402a459f8f63958dde80c256"
format_stamp: "Formatted at 2026-06-20 18:43:00 on dist-test-slave-wp6g"
server_key: "f41870c094b4ad2a687b6e2422282d80"
server_key_iv: "4e07ab1c3bd571f98d38c62e98951c7e"
server_key_version: "encryptionkey@0"
I20260620 18:43:00.943747 28445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260620 18:43:00.944600 28460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:00.944846 28445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:00.944955 28445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "69676d0a402a459f8f63958dde80c256"
format_stamp: "Formatted at 2026-06-20 18:43:00 on dist-test-slave-wp6g"
server_key: "f41870c094b4ad2a687b6e2422282d80"
server_key_iv: "4e07ab1c3bd571f98d38c62e98951c7e"
server_key_version: "encryptionkey@0"
I20260620 18:43:00.945034 28445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:00.959394 28445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:00.961906 28445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:00.962086 28445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:00.962350 28445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:00.962738 28445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:00.962791 28445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:00.962837 28445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:00.962865 28445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:00.969215 28445 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:36377
I20260620 18:43:00.969255 28573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:36377 every 8 connection(s)
I20260620 18:43:00.969594 28445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:43:00.975375 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 28445
I20260620 18:43:00.975493 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:43:00.975767 11241 external_mini_cluster.cc:1428] Setting key de325aeabe9e87004251440e080207aa
Jun 20 18:43:00 dist-test-slave-wp6g krb5kdc[28204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980980, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:00.980630 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:00.970397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:49457 (local address 127.10.250.126:38837)
0620 18:43:00.970480 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:00.970482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:00.970734 (+ 252us) server_negotiation.cc:407] Connection header received
0620 18:43:00.971295 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:00.971296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:00.971307 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:00.971365 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:00.972412 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.972966 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.974102 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.974317 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.974384 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:00.977405 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:00.977407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:00.977407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:00.977416 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:00.979007 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:00.979496 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:00.979498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:00.979498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:00.979566 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:00.979840 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:00.979841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:00.979842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:00.980003 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:00.980074 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:00.980383 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 18:43:00.980479 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:43:00.980763 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:00.970353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38837 (local address 127.10.250.66:49457)
0620 18:43:00.970559 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:00.970636 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:00.971178 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:00.971430 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:00.971434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:00.972051 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:00.972280 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:00.972286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.973089 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.973090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:00.973997 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:00.974000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.974400 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.974401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:00.974505 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:00.975172 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:00.975183 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:00.977265 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:00.979170 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:00.979178 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:00.979180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:00.979416 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:00.979641 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:00.979642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:00.979642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:00.979752 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:00.980074 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:00.980079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:00.980233 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 18:43:00.980326 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:43:00.980526 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260620 18:43:00.981256 28574 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38837
I20260620 18:43:00.981361 28574 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:00.981581 28574 heartbeater.cc:511] Master 127.10.250.126:38837 requested a full tablet report, sending...
I20260620 18:43:00.982070 28249 ts_manager.cc:194] Registered new tserver with Master: 69676d0a402a459f8f63958dde80c256 (127.10.250.66:36377)
I20260620 18:43:00.982430 28249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:49457
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:43:01.003955 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38837
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:01.090207 28579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:01.090497 28579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:01.090562 28579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:01.091974 28579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:01.092181 28579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:01.092240 28579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:43:01.094156 28579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:01.094483 28579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:01.094779 28579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:01.100350 28585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:01.100368 28584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:01.100508 28587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:01.100369 28579 server_base.cc:1034] running on GCE node
I20260620 18:43:01.100885 28579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:01.101162 28579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:01.102357 28579 hybrid_clock.cc:648] HybridClock initialized: now 1781980981102350 us; error 38 us; skew 500 ppm
Jun 20 18:43:01 dist-test-slave-wp6g krb5kdc[28204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980981, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:01.105228 28579 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:43:01.105779 28579 webserver.cc:460] Webserver started at http://127.10.250.67:33377/ using document root <none> and password file <none>
I20260620 18:43:01.106036 28579 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:01.106108 28579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:01.106278 28579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:01.107472 28579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "859890db48a646168c22ba51769df4c8"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-wp6g"
server_key: "fd40ff125a9abcc2170d1efc08f2349f"
server_key_iv: "5460fe29fbf7f7da368a795e14ea08a6"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.107858 28579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "859890db48a646168c22ba51769df4c8"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-wp6g"
server_key: "fd40ff125a9abcc2170d1efc08f2349f"
server_key_iv: "5460fe29fbf7f7da368a795e14ea08a6"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.109337 28579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:43:01.110193 28594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:01.110409 28579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:01.110514 28579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "859890db48a646168c22ba51769df4c8"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-wp6g"
server_key: "fd40ff125a9abcc2170d1efc08f2349f"
server_key_iv: "5460fe29fbf7f7da368a795e14ea08a6"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.110647 28579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:01.137272 28579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:01.139479 28579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:01.139647 28579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:01.139927 28579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:01.140268 28579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:01.140327 28579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:01.140369 28579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:01.140396 28579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:01.146817 28579 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:38621
I20260620 18:43:01.146900 28707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:38621 every 8 connection(s)
I20260620 18:43:01.147256 28579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:43:01.149652 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 28579
I20260620 18:43:01.149781 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:43:01.150012 11241 external_mini_cluster.cc:1428] Setting key d76ad53870b096e83d2734d622d81eb5
Jun 20 18:43:01 dist-test-slave-wp6g krb5kdc[28204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980981, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:01.157995 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.148310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:59689 (local address 127.10.250.126:38837)
0620 18:43:01.148360 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:01.148362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:01.148618 (+ 256us) server_negotiation.cc:407] Connection header received
0620 18:43:01.149330 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:01.149332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:01.149344 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:01.149404 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:01.150763 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.151382 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.152472 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.152687 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.152758 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.155222 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:01.155224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:01.155224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:01.155232 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:01.156685 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:01.157105 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:01.157105 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:01.157106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:01.157156 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:01.157374 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:01.157375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:01.157375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:01.157519 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:01.157580 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:01.157863 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 18:43:01.157927 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:43:01.158252 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.148319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38837 (local address 127.10.250.67:59689)
0620 18:43:01.148499 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:01.148541 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:01.149185 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:01.149503 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:01.149508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:01.150371 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:01.150685 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.150691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.151489 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.151490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.152369 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.152372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.152770 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.152771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.152879 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.153510 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:01.153519 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:01.155104 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:01.156771 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:01.156778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:01.156780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:01.157015 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:01.157216 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:01.157217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:01.157217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:01.157322 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:01.157577 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:01.157582 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:01.157749 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 18:43:01.157840 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 18:43:01.157968 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260620 18:43:01.158965 28708 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38837
I20260620 18:43:01.159082 28708 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:01.159291 28708 heartbeater.cc:511] Master 127.10.250.126:38837 requested a full tablet report, sending...
I20260620 18:43:01.159723 28249 ts_manager.cc:194] Registered new tserver with Master: 859890db48a646168c22ba51769df4c8 (127.10.250.67:38621)
I20260620 18:43:01.160112 28249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:59689
I20260620 18:43:01.161988 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:43:01.169955 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.163338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53184 (local address 127.10.250.126:38837)
0620 18:43:01.163417 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:01.163419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:01.163519 (+ 100us) server_negotiation.cc:407] Connection header received
0620 18:43:01.163559 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:01.163559 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:01.163569 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:01.163625 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:01.164154 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.164695 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.165412 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.165629 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.165691 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.167046 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:01.167047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:01.167048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:01.167055 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:01.168543 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:01.168950 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:01.168951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:01.168952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:01.169012 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:01.169233 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:01.169234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:01.169234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:01.169407 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:01.169523 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:01.169750 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 18:43:01.169845 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:43:01.171700 28249 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:43:01.173384 28249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:43:01.180896 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.176443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35316 (local address 127.10.250.67:38621)
0620 18:43:01.176644 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:01.176650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:01.176769 (+ 119us) server_negotiation.cc:407] Connection header received
0620 18:43:01.177291 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:01.177295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:01.177312 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:01.177420 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:01.177540 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.178264 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.179748 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.180431 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.180508 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.180568 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:01.180754 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 18:43:01.180804 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260620 18:43:01.180912 28719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.176258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:38621 (local address 127.0.0.1:35316)
0620 18:43:01.176696 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:01.176707 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:01.176794 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:01.177419 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:01.177420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:01.177423 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:01.177481 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.177483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.178387 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.178388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.179654 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.179657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.180522 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.180523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.180641 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.180656 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 18:43:01.180719 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 18:43:01.180798 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":91,"spinlock_wait_cycles":768,"thread_start_us":89,"threads_started":1}
I20260620 18:43:01.181010 28720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.176258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:36377 (local address 127.0.0.1:54086)
0620 18:43:01.176426 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:01.176474 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:01.176545 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:01.176894 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:01.176897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:01.176900 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:01.176980 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.176982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.178365 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.178365 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.179920 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.179923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.180787 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.180787 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.180848 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.180851 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:43:01.180902 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 18:43:01.180943 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":109,"thread_start_us":62,"threads_started":1}
I20260620 18:43:01.181285 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.176394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54086 (local address 127.10.250.66:36377)
0620 18:43:01.176692 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:01.176697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:01.176708 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:43:01.176752 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:01.176754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:01.176770 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:01.176883 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:01.177095 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.178242 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.180016 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.180685 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.180769 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.180822 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:01.181160 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 18:43:01.181205 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":98,"threads_started":1}
I20260620 18:43:01.181972 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.176458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:43827 (local address 127.0.0.1:57604)
0620 18:43:01.176732 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:01.176740 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:01.176786 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:01.177284 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:01.177285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:01.177287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:01.177334 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.177336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.178602 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.178602 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.179920 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.179923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.181736 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.181737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.181798 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.181803 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:43:01.181861 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 18:43:01.181902 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":7,"thread_start_us":65,"threads_started":1}
I20260620 18:43:01.182067 28508 tablet_service.cc:1467] Processing CreateTablet for tablet 9954128f53764a7cb8c64c8367d21f4b (DEFAULT_TABLE table=test-table [id=b2046cf46d1f4b2680419f2071e132fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:01.182240 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.176764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57604 (local address 127.10.250.65:43827)
0620 18:43:01.177006 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:01.177031 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:01.177062 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:43:01.177103 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:01.177104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:01.177120 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:01.177201 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:01.177414 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.178454 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.180858 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.181656 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.181727 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.181789 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:01.182111 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 18:43:01.182163 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":127,"threads_started":1}
I20260620 18:43:01.182488 28508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9954128f53764a7cb8c64c8367d21f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:01.182467 28642 tablet_service.cc:1467] Processing CreateTablet for tablet 9954128f53764a7cb8c64c8367d21f4b (DEFAULT_TABLE table=test-table [id=b2046cf46d1f4b2680419f2071e132fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:01.182860 28642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9954128f53764a7cb8c64c8367d21f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:01.183269 28374 tablet_service.cc:1467] Processing CreateTablet for tablet 9954128f53764a7cb8c64c8367d21f4b (DEFAULT_TABLE table=test-table [id=b2046cf46d1f4b2680419f2071e132fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:01.183604 28374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9954128f53764a7cb8c64c8367d21f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:01.185524 28725 tablet_bootstrap.cc:492] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256: Bootstrap starting.
I20260620 18:43:01.186347 28725 tablet_bootstrap.cc:654] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:01.186445 28726 tablet_bootstrap.cc:492] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8: Bootstrap starting.
I20260620 18:43:01.187041 28725 log.cc:824] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:01.187237 28727 tablet_bootstrap.cc:492] T 9954128f53764a7cb8c64c8367d21f4b P 48f8b0786a9541bfad6707f4701634ff: Bootstrap starting.
I20260620 18:43:01.187263 28726 tablet_bootstrap.cc:654] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:01.187733 28726 log.cc:824] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:01.187933 28727 tablet_bootstrap.cc:654] T 9954128f53764a7cb8c64c8367d21f4b P 48f8b0786a9541bfad6707f4701634ff: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:01.188410 28727 log.cc:824] T 9954128f53764a7cb8c64c8367d21f4b P 48f8b0786a9541bfad6707f4701634ff: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:01.188448 28726 tablet_bootstrap.cc:492] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8: No bootstrap required, opened a new log
I20260620 18:43:01.188520 28726 ts_tablet_manager.cc:1397] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:43:01.189112 28727 tablet_bootstrap.cc:492] T 9954128f53764a7cb8c64c8367d21f4b P 48f8b0786a9541bfad6707f4701634ff: No bootstrap required, opened a new log
I20260620 18:43:01.189191 28727 ts_tablet_manager.cc:1397] T 9954128f53764a7cb8c64c8367d21f4b P 48f8b0786a9541bfad6707f4701634ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:43:01.190507 28726 raft_consensus.cc:348] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859890db48a646168c22ba51769df4c8" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38621 } } peers { permanent_uuid: "69676d0a402a459f8f63958dde80c256" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36377 } } peers { permanent_uuid: "48f8b0786a9541bfad6707f4701634ff" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43827 } }
I20260620 18:43:01.190744 28726 raft_consensus.cc:374] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:01.190819 28726 raft_consensus.cc:729] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859890db48a646168c22ba51769df4c8, State: Initialized, Role: FOLLOWER
I20260620 18:43:01.190925 28725 tablet_bootstrap.cc:492] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256: No bootstrap required, opened a new log
I20260620 18:43:01.191005 28725 ts_tablet_manager.cc:1397] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 18:43:01.190958 28726 consensus_queue.cc:260] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859890db48a646168c22ba51769df4c8" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38621 } } peers { permanent_uuid: "69676d0a402a459f8f63958dde80c256" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36377 } } peers { permanent_uuid: "48f8b0786a9541bfad6707f4701634ff" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43827 } }
I20260620 18:43:01.190930 28727 raft_consensus.cc:348] T 9954128f53764a7cb8c64c8367d21f4b P 48f8b0786a9541bfad6707f4701634ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859890db48a646168c22ba51769df4c8" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38621 } } peers { permanent_uuid: "69676d0a402a459f8f63958dde80c256" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36377 } } peers { permanent_uuid: "48f8b0786a9541bfad6707f4701634ff" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43827 } }
I20260620 18:43:01.191103 28727 raft_consensus.cc:374] T 9954128f53764a7cb8c64c8367d21f4b P 48f8b0786a9541bfad6707f4701634ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:01.191139 28727 raft_consensus.cc:729] T 9954128f53764a7cb8c64c8367d21f4b P 48f8b0786a9541bfad6707f4701634ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f8b0786a9541bfad6707f4701634ff, State: Initialized, Role: FOLLOWER
I20260620 18:43:01.191262 28726 ts_tablet_manager.cc:1428] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 18:43:01.191319 28727 consensus_queue.cc:260] T 9954128f53764a7cb8c64c8367d21f4b P 48f8b0786a9541bfad6707f4701634ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859890db48a646168c22ba51769df4c8" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38621 } } peers { permanent_uuid: "69676d0a402a459f8f63958dde80c256" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36377 } } peers { permanent_uuid: "48f8b0786a9541bfad6707f4701634ff" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43827 } }
I20260620 18:43:01.191409 28708 heartbeater.cc:503] Master 127.10.250.126:38837 was elected leader, sending a full tablet report...
I20260620 18:43:01.191604 28727 ts_tablet_manager.cc:1428] T 9954128f53764a7cb8c64c8367d21f4b P 48f8b0786a9541bfad6707f4701634ff: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 18:43:01.191663 28440 heartbeater.cc:503] Master 127.10.250.126:38837 was elected leader, sending a full tablet report...
I20260620 18:43:01.193007 28725 raft_consensus.cc:348] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859890db48a646168c22ba51769df4c8" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38621 } } peers { permanent_uuid: "69676d0a402a459f8f63958dde80c256" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36377 } } peers { permanent_uuid: "48f8b0786a9541bfad6707f4701634ff" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43827 } }
I20260620 18:43:01.193140 28725 raft_consensus.cc:374] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:01.193161 28725 raft_consensus.cc:729] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69676d0a402a459f8f63958dde80c256, State: Initialized, Role: FOLLOWER
I20260620 18:43:01.193279 28725 consensus_queue.cc:260] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859890db48a646168c22ba51769df4c8" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38621 } } peers { permanent_uuid: "69676d0a402a459f8f63958dde80c256" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36377 } } peers { permanent_uuid: "48f8b0786a9541bfad6707f4701634ff" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43827 } }
I20260620 18:43:01.193562 28725 ts_tablet_manager.cc:1428] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:43:01.193585 28574 heartbeater.cc:503] Master 127.10.250.126:38837 was elected leader, sending a full tablet report...
W20260620 18:43:01.220194 28575 tablet.cc:2369] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:43:01.304864 28441 tablet.cc:2369] T 9954128f53764a7cb8c64c8367d21f4b P 48f8b0786a9541bfad6707f4701634ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:01.395150 28733 raft_consensus.cc:482] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:43:01.395299 28733 raft_consensus.cc:504] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859890db48a646168c22ba51769df4c8" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38621 } } peers { permanent_uuid: "69676d0a402a459f8f63958dde80c256" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36377 } } peers { permanent_uuid: "48f8b0786a9541bfad6707f4701634ff" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43827 } }
I20260620 18:43:01.395766 28733 leader_election.cc:288] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 859890db48a646168c22ba51769df4c8 (127.10.250.67:38621), 48f8b0786a9541bfad6707f4701634ff (127.10.250.65:43827)
W20260620 18:43:01.397864 28709 tablet.cc:2369] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:01.399097 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.396108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:52981 (local address 127.10.250.67:38621)
0620 18:43:01.396265 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:01.396267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:01.396275 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:43:01.396317 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:01.396318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:01.396328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:01.396380 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:01.396620 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.397094 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.398280 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.398722 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.398810 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.398819 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:01.398982 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 18:43:01.399021 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260620 18:43:01.399099 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.396253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:59765 (local address 127.10.250.65:43827)
0620 18:43:01.396322 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:01.396324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:01.396465 (+ 141us) server_negotiation.cc:407] Connection header received
0620 18:43:01.396531 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:01.396532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:01.396540 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:01.396581 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:01.396752 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.397215 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.398283 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.398733 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.398827 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.398836 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:01.398987 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 18:43:01.399023 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 18:43:01.399138 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.396026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:38621 (local address 127.10.250.66:52981)
0620 18:43:01.396106 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:01.396119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:01.396169 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:01.396399 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:01.396400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:01.396402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:01.396479 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.396481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.397192 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.397194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.398180 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.398183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.398812 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.398813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.398896 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.398900 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:43:01.398974 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 18:43:01.399019 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 18:43:01.399154 28734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.396144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:43827 (local address 127.10.250.66:59765)
0620 18:43:01.396373 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:01.396384 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:01.396465 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:01.396606 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:01.396607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:01.396608 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:01.396656 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.396657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.397301 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.397302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.398192 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.398195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.398827 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.398829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.398903 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.398909 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:43:01.398982 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 18:43:01.399022 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":137,"threads_started":1}
I20260620 18:43:01.399628 28662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9954128f53764a7cb8c64c8367d21f4b" candidate_uuid: "69676d0a402a459f8f63958dde80c256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859890db48a646168c22ba51769df4c8" is_pre_election: true
I20260620 18:43:01.399799 28662 raft_consensus.cc:2454] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69676d0a402a459f8f63958dde80c256 in term 0.
I20260620 18:43:01.399734 28394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9954128f53764a7cb8c64c8367d21f4b" candidate_uuid: "69676d0a402a459f8f63958dde80c256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f8b0786a9541bfad6707f4701634ff" is_pre_election: true
I20260620 18:43:01.399924 28394 raft_consensus.cc:2454] T 9954128f53764a7cb8c64c8367d21f4b P 48f8b0786a9541bfad6707f4701634ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69676d0a402a459f8f63958dde80c256 in term 0.
I20260620 18:43:01.400048 28464 leader_election.cc:302] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69676d0a402a459f8f63958dde80c256, 859890db48a646168c22ba51769df4c8; no voters:
I20260620 18:43:01.400190 28733 raft_consensus.cc:2781] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:43:01.400247 28733 raft_consensus.cc:482] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:43:01.400278 28733 raft_consensus.cc:3037] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:01.401000 28733 raft_consensus.cc:504] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859890db48a646168c22ba51769df4c8" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38621 } } peers { permanent_uuid: "69676d0a402a459f8f63958dde80c256" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36377 } } peers { permanent_uuid: "48f8b0786a9541bfad6707f4701634ff" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43827 } }
I20260620 18:43:01.401144 28733 leader_election.cc:288] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [CANDIDATE]: Term 1 election: Requested vote from peers 859890db48a646168c22ba51769df4c8 (127.10.250.67:38621), 48f8b0786a9541bfad6707f4701634ff (127.10.250.65:43827)
I20260620 18:43:01.401363 28662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9954128f53764a7cb8c64c8367d21f4b" candidate_uuid: "69676d0a402a459f8f63958dde80c256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859890db48a646168c22ba51769df4c8"
I20260620 18:43:01.401363 28394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9954128f53764a7cb8c64c8367d21f4b" candidate_uuid: "69676d0a402a459f8f63958dde80c256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f8b0786a9541bfad6707f4701634ff"
I20260620 18:43:01.401458 28662 raft_consensus.cc:3037] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:01.401458 28394 raft_consensus.cc:3037] T 9954128f53764a7cb8c64c8367d21f4b P 48f8b0786a9541bfad6707f4701634ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:01.402401 28662 raft_consensus.cc:2454] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69676d0a402a459f8f63958dde80c256 in term 1.
I20260620 18:43:01.402401 28394 raft_consensus.cc:2454] T 9954128f53764a7cb8c64c8367d21f4b P 48f8b0786a9541bfad6707f4701634ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69676d0a402a459f8f63958dde80c256 in term 1.
I20260620 18:43:01.402601 28464 leader_election.cc:302] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f8b0786a9541bfad6707f4701634ff, 69676d0a402a459f8f63958dde80c256; no voters:
I20260620 18:43:01.402737 28733 raft_consensus.cc:2781] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:01.402882 28733 raft_consensus.cc:686] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [term 1 LEADER]: Becoming Leader. State: Replica: 69676d0a402a459f8f63958dde80c256, State: Running, Role: LEADER
I20260620 18:43:01.402989 28733 consensus_queue.cc:237] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859890db48a646168c22ba51769df4c8" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38621 } } peers { permanent_uuid: "69676d0a402a459f8f63958dde80c256" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36377 } } peers { permanent_uuid: "48f8b0786a9541bfad6707f4701634ff" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43827 } }
I20260620 18:43:01.403867 28248 catalog_manager.cc:5495] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69676d0a402a459f8f63958dde80c256 (127.10.250.66). New cstate: current_term: 1 leader_uuid: "69676d0a402a459f8f63958dde80c256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859890db48a646168c22ba51769df4c8" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69676d0a402a459f8f63958dde80c256" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48f8b0786a9541bfad6707f4701634ff" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43827 } health_report { overall_health: UNKNOWN } } }
I20260620 18:43:01.411914 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.408905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54090 (local address 127.10.250.66:36377)
0620 18:43:01.409075 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:01.409077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:01.409085 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:43:01.409122 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:01.409123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:01.409156 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:01.409221 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:01.409457 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.410044 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.411063 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.411265 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.411331 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.411510 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:01.411650 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:01.411718 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:01.411827 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 18:43:01.411848 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260620 18:43:01.414156 28394 raft_consensus.cc:1261] T 9954128f53764a7cb8c64c8367d21f4b P 48f8b0786a9541bfad6707f4701634ff [term 1 FOLLOWER]: Refusing update from remote peer 69676d0a402a459f8f63958dde80c256: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:01.414166 28662 raft_consensus.cc:1261] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8 [term 1 FOLLOWER]: Refusing update from remote peer 69676d0a402a459f8f63958dde80c256: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:01.414615 28733 consensus_queue.cc:1035] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48f8b0786a9541bfad6707f4701634ff" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:01.414796 28735 consensus_queue.cc:1035] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "859890db48a646168c22ba51769df4c8" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 38621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:01.417739 28738 mvcc.cc:205] Tried to move back new op lower bound from 7298994099869069312 to 7298994099827187712. Current Snapshot: MvccSnapshot[applied={T|T < 7298994099869069312}]
I20260620 18:43:01.417793 28740 mvcc.cc:205] Tried to move back new op lower bound from 7298994099869069312 to 7298994099827187712. Current Snapshot: MvccSnapshot[applied={T|T < 7298994099869069312}]
I20260620 18:43:01.418351 28739 mvcc.cc:205] Tried to move back new op lower bound from 7298994099869069312 to 7298994099827187712. Current Snapshot: MvccSnapshot[applied={T|T < 7298994099869069312}]
I20260620 18:43:01.420320 28247 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:01.420439 28247 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:01.421265 28247 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 877ff278abcd4d4ebba04f3d865ec59b: Sending DeleteTablet for 3 replicas of tablet 9954128f53764a7cb8c64c8367d21f4b
I20260620 18:43:01.421710 28508 tablet_service.cc:1514] Processing DeleteTablet for tablet 9954128f53764a7cb8c64c8367d21f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:01 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:54086
I20260620 18:43:01.421732 28642 tablet_service.cc:1514] Processing DeleteTablet for tablet 9954128f53764a7cb8c64c8367d21f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:01 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:35316
I20260620 18:43:01.421984 28374 tablet_service.cc:1514] Processing DeleteTablet for tablet 9954128f53764a7cb8c64c8367d21f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:01 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:57604
I20260620 18:43:01.422303 28750 tablet_replica.cc:330] T 9954128f53764a7cb8c64c8367d21f4b P 48f8b0786a9541bfad6707f4701634ff: stopping tablet replica
I20260620 18:43:01.422317 28749 tablet_replica.cc:330] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8: stopping tablet replica
I20260620 18:43:01.422427 28749 raft_consensus.cc:2229] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:01.422546 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 28311
I20260620 18:43:01.422618 28749 raft_consensus.cc:2258] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:01.424326 28749 ts_tablet_manager.cc:1905] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:01.426101 28749 ts_tablet_manager.cc:1918] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:01.426180 28749 log.cc:1196] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/9954128f53764a7cb8c64c8367d21f4b
I20260620 18:43:01.426470 28749 ts_tablet_manager.cc:1939] T 9954128f53764a7cb8c64c8367d21f4b P 859890db48a646168c22ba51769df4c8: Deleting consensus metadata
I20260620 18:43:01.423900 28748 tablet_replica.cc:330] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256: stopping tablet replica
I20260620 18:43:01.426774 28748 raft_consensus.cc:2229] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:43:01.426923 28748 raft_consensus.cc:2258] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:01.427399 28748 ts_tablet_manager.cc:1905] T 9954128f53764a7cb8c64c8367d21f4b P 69676d0a402a459f8f63958dde80c256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:01.427814 28236 catalog_manager.cc:4841] TS 859890db48a646168c22ba51769df4c8 (127.10.250.67:38621): tablet 9954128f53764a7cb8c64c8367d21f4b (table test-table [id=b2046cf46d1f4b2680419f2071e132fb]) successfully deleted
W20260620 18:43:01.428011 28236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:43827 (error 108)
I20260620 18:43:01.428802 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.428669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:43827 (local address 127.0.0.1:57620)
0620 18:43:01.428721 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:01.428755 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:43827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 18:43:01.428848 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 28445
W20260620 18:43:01.429055 28236 catalog_manager.cc:4584] TS 48f8b0786a9541bfad6707f4701634ff (127.10.250.65:43827): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9954128f53764a7cb8c64c8367d21f4b: Network error: Client connection negotiation failed: client connection to 127.10.250.65:43827: connect: Connection refused (error 111)
I20260620 18:43:01.434615 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 28579
I20260620 18:43:01.434849 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.434632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:36377 (local address 127.0.0.1:54094)
0620 18:43:01.434766 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:01.434800 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.66:36377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 18:43:01.439532 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 28217
2026-06-20T18:43:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2160 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T18:43:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:01Z Disabled control of system clock
I20260620 18:43:01.468482 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:42153
--webserver_interface=127.10.250.126
--webserver_port=0
--builtin_ntp_servers=127.10.250.84:33401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:42153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:43:01.553782 28755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:01.554029 28755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:01.554052 28755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:01.555382 28755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:01.555435 28755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:01.555449 28755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:01.555462 28755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:01.555475 28755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:01.556847 28755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:33401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:42153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:42153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:01.557065 28755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:01.557322 28755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:01.562366 28761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:01.562373 28763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:01.562587 28755 server_base.cc:1034] running on GCE node
W20260620 18:43:01.562373 28760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:01.563138 28755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:01.563515 28755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:01.564742 28755 hybrid_clock.cc:648] HybridClock initialized: now 1781980981564711 us; error 39 us; skew 500 ppm
I20260620 18:43:01.566254 28755 webserver.cc:460] Webserver started at http://127.10.250.126:34373/ using document root <none> and password file <none>
I20260620 18:43:01.566587 28755 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:01.566673 28755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:01.566819 28755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:01.567967 28755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "5cdd7b6f4885476288d2aff528bd1f8a"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-wp6g"
server_key: "f2d00e9db6fd870fb24b1ec6758d534b"
server_key_iv: "d81fe70188b5b19a3bfaab1bde70a839"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.568365 28755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "5cdd7b6f4885476288d2aff528bd1f8a"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-wp6g"
server_key: "f2d00e9db6fd870fb24b1ec6758d534b"
server_key_iv: "d81fe70188b5b19a3bfaab1bde70a839"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.569859 28755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:43:01.570892 28769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:01.571095 28755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:01.571216 28755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "5cdd7b6f4885476288d2aff528bd1f8a"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-wp6g"
server_key: "f2d00e9db6fd870fb24b1ec6758d534b"
server_key_iv: "d81fe70188b5b19a3bfaab1bde70a839"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.571308 28755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:01.591133 28755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:01.591540 28755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:01.591730 28755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:01.596611 28755 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:42153
I20260620 18:43:01.596673 28821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:42153 every 8 connection(s)
I20260620 18:43:01.597056 28755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:01.598357 28822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:01.600878 28822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a: Bootstrap starting.
I20260620 18:43:01.601691 28822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:01.602039 28822 log.cc:824] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:01.603008 28822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a: No bootstrap required, opened a new log
I20260620 18:43:01.603917 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 28755
I20260620 18:43:01.604003 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:01.604221 11241 external_mini_cluster.cc:1428] Setting key d8fa24b79cd7ad25986134ec5fa77961
I20260620 18:43:01.605329 28822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdd7b6f4885476288d2aff528bd1f8a" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42153 } }
I20260620 18:43:01.605540 28822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:01.605592 28822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cdd7b6f4885476288d2aff528bd1f8a, State: Initialized, Role: FOLLOWER
I20260620 18:43:01.605780 28822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdd7b6f4885476288d2aff528bd1f8a" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42153 } }
I20260620 18:43:01.605857 28822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:01.605880 28822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:01.605952 28822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:01.606685 28822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdd7b6f4885476288d2aff528bd1f8a" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42153 } }
I20260620 18:43:01.607000 28822 leader_election.cc:302] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cdd7b6f4885476288d2aff528bd1f8a; no voters:
I20260620 18:43:01.607270 28822 leader_election.cc:288] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:01.607312 28827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:01.607542 28827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [term 1 LEADER]: Becoming Leader. State: Replica: 5cdd7b6f4885476288d2aff528bd1f8a, State: Running, Role: LEADER
I20260620 18:43:01.607686 28822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:01.607702 28827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdd7b6f4885476288d2aff528bd1f8a" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42153 } }
I20260620 18:43:01.608290 28828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cdd7b6f4885476288d2aff528bd1f8a. Latest consensus state: current_term: 1 leader_uuid: "5cdd7b6f4885476288d2aff528bd1f8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdd7b6f4885476288d2aff528bd1f8a" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42153 } } }
I20260620 18:43:01.608392 28828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:01.608659 28829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cdd7b6f4885476288d2aff528bd1f8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdd7b6f4885476288d2aff528bd1f8a" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 42153 } } }
I20260620 18:43:01.609025 28829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:01.609229 28834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:01.609918 28834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:01.611178 28826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.604874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.10.250.126:42153)
0620 18:43:01.605110 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:01.605118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:01.605147 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:43:01.605278 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:01.605284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:01.605344 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:01.605471 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:01.605737 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.607252 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.608354 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.609011 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.609150 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.610242 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:01.610247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:01.610247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:01.610647 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:01.610689 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:01.610703 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:01.610787 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:01.610872 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 18:43:01.610965 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":110,"threads_started":1}
I20260620 18:43:01.613379 28834 catalog_manager.cc:1347] Generated new cluster ID: d9b36979c91744279283d2be66681adf
I20260620 18:43:01.613476 28834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:01.616852 28834 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:01.617728 28834 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:01.625620 28834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a: Generated new TSK 0
I20260620 18:43:01.625901 28834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:01.630574 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:42153
--builtin_ntp_servers=127.10.250.84:33401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:43:01.720177 28846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:01.720520 28846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:01.720568 28846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:01.722035 28846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:01.722133 28846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:01.722182 28846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:43:01.723842 28846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:33401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:01.724159 28846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:01.724509 28846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:01.731098 28854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:01.731210 28846 server_base.cc:1034] running on GCE node
W20260620 18:43:01.731570 28852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:01.731106 28851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:01.731853 28846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:01.732147 28846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:01.733330 28846 hybrid_clock.cc:648] HybridClock initialized: now 1781980981733318 us; error 35 us; skew 500 ppm
I20260620 18:43:01.734838 28846 webserver.cc:460] Webserver started at http://127.10.250.65:44967/ using document root <none> and password file <none>
I20260620 18:43:01.735110 28846 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:01.735184 28846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:01.735363 28846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:01.736469 28846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "0729e58bc11546bdb97cced07a39f5dd"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-wp6g"
server_key: "33c6842474b107eaad14a5f392289444"
server_key_iv: "177cbb766283f29b9090da2ec7d534c8"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.736843 28846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "0729e58bc11546bdb97cced07a39f5dd"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-wp6g"
server_key: "33c6842474b107eaad14a5f392289444"
server_key_iv: "177cbb766283f29b9090da2ec7d534c8"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.738265 28846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:43:01.739229 28860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:01.739414 28846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:43:01.739521 28846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "0729e58bc11546bdb97cced07a39f5dd"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-wp6g"
server_key: "33c6842474b107eaad14a5f392289444"
server_key_iv: "177cbb766283f29b9090da2ec7d534c8"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.739609 28846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:01.752918 28846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:01.753305 28846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:01.753476 28846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:01.753732 28846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:01.754084 28846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:01.754141 28846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:01.754189 28846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:01.754215 28846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:01.760828 28846 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:36715
I20260620 18:43:01.760872 28973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:36715 every 8 connection(s)
I20260620 18:43:01.761219 28846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:43:01.766453 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 28846
I20260620 18:43:01.766604 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:43:01.766816 11241 external_mini_cluster.cc:1428] Setting key 19ecae0e5e9b2dc0873e8fd9b802be6e
I20260620 18:43:01.767093 28826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.761997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:48767 (local address 127.10.250.126:42153)
0620 18:43:01.762067 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:01.762070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:01.762387 (+ 317us) server_negotiation.cc:407] Connection header received
0620 18:43:01.763030 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:01.763032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:01.763045 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:01.763101 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:01.763826 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.764347 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.765358 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.765541 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.765593 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.766691 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:01.766692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:01.766693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:01.766700 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:01.766718 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:01.766722 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:01.766751 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:01.766948 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 18:43:01.766984 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:43:01.767560 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.761992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:42153 (local address 127.10.250.65:48767)
0620 18:43:01.762261 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:01.762299 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:01.762923 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:01.763165 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:01.763168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:01.763177 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:01.763727 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.763732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.764431 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.764431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.765281 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.765284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.765614 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.765615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.765764 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.766450 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:01.766451 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:01.766500 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:01.766570 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:01.766793 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:01.766799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:01.766802 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:43:01.767246 (+ 444us) client_negotiation.cc:240] Negotiation successful
0620 18:43:01.767370 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":118,"threads_started":1}
I20260620 18:43:01.768275 28974 heartbeater.cc:347] Connected to a master server at 127.10.250.126:42153
I20260620 18:43:01.768389 28974 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:01.768602 28974 heartbeater.cc:511] Master 127.10.250.126:42153 requested a full tablet report, sending...
I20260620 18:43:01.769275 28786 ts_manager.cc:194] Registered new tserver with Master: 0729e58bc11546bdb97cced07a39f5dd (127.10.250.65:36715)
I20260620 18:43:01.770319 28786 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.250.65:48767
I20260620 18:43:01.771607 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:42153
--builtin_ntp_servers=127.10.250.84:33401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:43:01.860088 28977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:01.860412 28977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:01.860457 28977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:01.861995 28977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:01.862087 28977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:01.862143 28977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:43:01.863791 28977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:33401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:01.864137 28977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:01.864490 28977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:01.870355 28982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:01.870390 28985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:01.870412 28977 server_base.cc:1034] running on GCE node
W20260620 18:43:01.870390 28983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:01.870878 28977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:01.871182 28977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:01.872383 28977 hybrid_clock.cc:648] HybridClock initialized: now 1781980981872368 us; error 49 us; skew 500 ppm
I20260620 18:43:01.874125 28977 webserver.cc:460] Webserver started at http://127.10.250.66:33493/ using document root <none> and password file <none>
I20260620 18:43:01.874424 28977 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:01.874495 28977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:01.874696 28977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:01.875931 28977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "79b5a564434145deb09c4166b8761119"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-wp6g"
server_key: "0c678dbeb90c403e04f6e2d11148e5ad"
server_key_iv: "3f2550137af9c50fdb752fef7320b546"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.876320 28977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "79b5a564434145deb09c4166b8761119"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-wp6g"
server_key: "0c678dbeb90c403e04f6e2d11148e5ad"
server_key_iv: "3f2550137af9c50fdb752fef7320b546"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.877920 28977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:43:01.879001 28991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:01.879330 28977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:01.879464 28977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "79b5a564434145deb09c4166b8761119"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-wp6g"
server_key: "0c678dbeb90c403e04f6e2d11148e5ad"
server_key_iv: "3f2550137af9c50fdb752fef7320b546"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.879556 28977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:01.896210 28977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:01.896639 28977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:01.896819 28977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:01.897100 28977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:01.897475 28977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:01.897531 28977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:01.897573 28977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:01.897603 28977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:01.904501 28977 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:35047
I20260620 18:43:01.904543 29104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:35047 every 8 connection(s)
I20260620 18:43:01.904954 28977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:43:01.908490 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 28977
I20260620 18:43:01.908591 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:43:01.908880 11241 external_mini_cluster.cc:1428] Setting key 264da79493266a142edcc8fb3b62cf87
I20260620 18:43:01.910140 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:42153
--builtin_ntp_servers=127.10.250.84:33401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:43:01.912195 28826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.905895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:54933 (local address 127.10.250.126:42153)
0620 18:43:01.905973 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:01.905975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:01.906236 (+ 261us) server_negotiation.cc:407] Connection header received
0620 18:43:01.906852 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:01.906853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:01.906864 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:01.906931 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:01.907702 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.908702 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.910131 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.910467 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.910589 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.911785 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:01.911787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:01.911787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:01.911793 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:01.911814 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:01.911820 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:01.911893 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:01.912060 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 18:43:01.912097 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:43:01.912320 29107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.905820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:42153 (local address 127.10.250.66:54933)
0620 18:43:01.906025 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:01.906078 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:01.906702 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:01.906995 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:01.906998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:01.907007 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:01.907604 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.907610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.908894 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.908895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.910005 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.910008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.910624 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.910625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.910759 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.911457 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:01.911457 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:01.911507 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:01.911539 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:01.911913 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:01.911920 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:01.911924 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:43:01.911992 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 18:43:01.912087 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260620 18:43:01.912967 29105 heartbeater.cc:347] Connected to a master server at 127.10.250.126:42153
I20260620 18:43:01.913062 29105 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:01.913281 29105 heartbeater.cc:511] Master 127.10.250.126:42153 requested a full tablet report, sending...
I20260620 18:43:01.913719 28786 ts_manager.cc:194] Registered new tserver with Master: 79b5a564434145deb09c4166b8761119 (127.10.250.66:35047)
I20260620 18:43:01.914111 28786 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.250.66:54933
W20260620 18:43:01.998004 29108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:01.998287 29108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:01.998330 29108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:01.999691 29108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:01.999787 29108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:01.999835 29108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:43:02.001345 29108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:33401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:02.001688 29108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:02.002027 29108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:02.009522 29114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:02.009572 29108 server_base.cc:1034] running on GCE node
W20260620 18:43:02.009585 29116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:02.009929 29113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:02.010318 29108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:02.010739 29108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:02.011962 29108 hybrid_clock.cc:648] HybridClock initialized: now 1781980982011919 us; error 68 us; skew 500 ppm
I20260620 18:43:02.013515 29108 webserver.cc:460] Webserver started at http://127.10.250.67:36113/ using document root <none> and password file <none>
I20260620 18:43:02.013813 29108 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:02.013870 29108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:02.014016 29108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:02.015313 29108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "531634ef874543e885d6b5913f34d182"
format_stamp: "Formatted at 2026-06-20 18:43:02 on dist-test-slave-wp6g"
server_key: "638f1ace9aa44ac2cec8ddb499e77217"
server_key_iv: "4ee7458f1412719b552c22db75ecb14a"
server_key_version: "encryptionkey@0"
I20260620 18:43:02.015700 29108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "531634ef874543e885d6b5913f34d182"
format_stamp: "Formatted at 2026-06-20 18:43:02 on dist-test-slave-wp6g"
server_key: "638f1ace9aa44ac2cec8ddb499e77217"
server_key_iv: "4ee7458f1412719b552c22db75ecb14a"
server_key_version: "encryptionkey@0"
I20260620 18:43:02.017282 29108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:43:02.018390 29122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:02.018635 29108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:02.018774 29108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "531634ef874543e885d6b5913f34d182"
format_stamp: "Formatted at 2026-06-20 18:43:02 on dist-test-slave-wp6g"
server_key: "638f1ace9aa44ac2cec8ddb499e77217"
server_key_iv: "4ee7458f1412719b552c22db75ecb14a"
server_key_version: "encryptionkey@0"
I20260620 18:43:02.018872 29108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:02.032940 29108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:02.033326 29108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:02.033493 29108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:02.033739 29108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:02.034111 29108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:02.034169 29108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:02.034210 29108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:02.034257 29108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:02.040047 29108 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:33727
I20260620 18:43:02.040091 29235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:33727 every 8 connection(s)
I20260620 18:43:02.040467 29108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:43:02.046571 28826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.041239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:55291 (local address 127.10.250.126:42153)
0620 18:43:02.041306 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.041308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.041565 (+ 257us) server_negotiation.cc:407] Connection header received
0620 18:43:02.042215 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.042216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.042230 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.042291 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:02.043112 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.043648 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.044703 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.044920 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.045011 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.046210 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:02.046211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:02.046211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:02.046216 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:02.046235 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:02.046240 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:02.046313 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.046419 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.046453 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:43:02.046682 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 29108
I20260620 18:43:02.046806 29238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.041256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:42153 (local address 127.10.250.67:55291)
0620 18:43:02.041437 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:02.041474 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:02.042094 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:02.042366 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:02.042370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:02.042378 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:02.043005 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.043011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.043753 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.043754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.044597 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.044599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.045026 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.045027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.045176 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.046022 (+ 846us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:02.046023 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:02.046055 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:02.046080 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:02.046312 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:02.046320 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:02.046323 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:43:02.046420 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:43:02.046554 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260620 18:43:02.046826 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:43:02.047039 11241 external_mini_cluster.cc:1428] Setting key 49a530e4b08e60e8e4e2f79eb3cd583d
I20260620 18:43:02.047304 29236 heartbeater.cc:347] Connected to a master server at 127.10.250.126:42153
I20260620 18:43:02.047418 29236 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:02.047636 29236 heartbeater.cc:511] Master 127.10.250.126:42153 requested a full tablet report, sending...
I20260620 18:43:02.048131 28786 ts_manager.cc:194] Registered new tserver with Master: 531634ef874543e885d6b5913f34d182 (127.10.250.67:33727)
I20260620 18:43:02.048527 28786 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.250.67:55291
I20260620 18:43:02.059314 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:43:02.064078 28826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.060803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.10.250.126:42153)
0620 18:43:02.060872 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.060874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.060979 (+ 105us) server_negotiation.cc:407] Connection header received
0620 18:43:02.061040 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.061041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.061052 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.061100 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:02.061262 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.061746 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.062499 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.062799 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.062872 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.063764 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:02.063766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:02.063766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:02.063772 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:02.063792 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:02.063797 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:02.063850 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.063949 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.063982 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 18:43:02.066376 28786 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:43:02.068262 28786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:43:02.075176 29245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.070741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:36715 (local address 127.0.0.1:40752)
0620 18:43:02.071068 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:02.071077 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:02.071119 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:02.071489 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:02.071490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:02.071493 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:02.071569 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.071571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.072931 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.072932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.073975 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.073977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.074841 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.074842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.074924 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.074938 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 18:43:02.075006 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 18:43:02.075064 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":114,"threads_started":1}
I20260620 18:43:02.075294 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.070977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:33727 (local address 127.0.0.1:36882)
0620 18:43:02.071100 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:02.071112 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:02.071381 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:02.071634 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:02.071635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:02.071637 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:02.071667 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.071669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.072962 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.072963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.073854 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.073856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.074966 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.074967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.075048 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.075052 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:43:02.075110 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 18:43:02.075158 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":27,"thread_start_us":73,"threads_started":1}
I20260620 18:43:02.075407 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.071231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.10.250.67:33727)
0620 18:43:02.071414 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.071418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.071428 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:43:02.071473 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.071475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.071490 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.071564 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:02.071743 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.072877 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.073964 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.074748 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.074819 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.074915 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.075146 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.075187 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260620 18:43:02.075539 29244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.070741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:35047 (local address 127.0.0.1:50604)
0620 18:43:02.070945 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:02.070994 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:02.071067 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:02.071477 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:02.071480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:02.071482 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:02.071559 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.071561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.072556 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.072557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.073827 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.073829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.074953 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.074954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.075048 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.075054 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:43:02.075106 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 18:43:02.075157 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":14,"thread_start_us":90,"threads_started":1}
I20260620 18:43:02.075572 29246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.070840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.10.250.66:35047)
0620 18:43:02.071228 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.071232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.071243 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:43:02.071287 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.071289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.071305 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.071383 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:02.071707 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.072453 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.073916 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.074745 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.074819 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.074916 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.075204 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.075422 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":89,"threads_started":1}
I20260620 18:43:02.075910 29247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.070905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40752 (local address 127.10.250.65:36715)
0620 18:43:02.071227 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.071231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.071240 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:43:02.071279 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.071280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.071292 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.071384 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:02.071685 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.072835 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.074055 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.074744 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.074838 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.074931 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.075695 (+ 764us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.075759 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":79,"threads_started":1}
I20260620 18:43:02.076136 29170 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbc56c7b3ad456fb758de777c488333 (DEFAULT_TABLE table=test-table [id=14a2ec73de6f4ee38be6b0797b396d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:02.076532 29170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbc56c7b3ad456fb758de777c488333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:02.076471 29039 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbc56c7b3ad456fb758de777c488333 (DEFAULT_TABLE table=test-table [id=14a2ec73de6f4ee38be6b0797b396d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:02.076508 28908 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbc56c7b3ad456fb758de777c488333 (DEFAULT_TABLE table=test-table [id=14a2ec73de6f4ee38be6b0797b396d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:02.076795 29039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbc56c7b3ad456fb758de777c488333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:02.076853 28908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbc56c7b3ad456fb758de777c488333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:02.080116 29250 tablet_bootstrap.cc:492] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119: Bootstrap starting.
I20260620 18:43:02.080636 29252 tablet_bootstrap.cc:492] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182: Bootstrap starting.
I20260620 18:43:02.081172 29250 tablet_bootstrap.cc:654] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:02.081357 29252 tablet_bootstrap.cc:654] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:02.081540 29250 log.cc:824] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:02.081672 29252 log.cc:824] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:02.081995 29251 tablet_bootstrap.cc:492] T 7dbc56c7b3ad456fb758de777c488333 P 0729e58bc11546bdb97cced07a39f5dd: Bootstrap starting.
I20260620 18:43:02.082597 29252 tablet_bootstrap.cc:492] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182: No bootstrap required, opened a new log
I20260620 18:43:02.082679 29252 ts_tablet_manager.cc:1397] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:43:02.082774 29251 tablet_bootstrap.cc:654] T 7dbc56c7b3ad456fb758de777c488333 P 0729e58bc11546bdb97cced07a39f5dd: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:02.082855 29250 tablet_bootstrap.cc:492] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119: No bootstrap required, opened a new log
I20260620 18:43:02.082931 29250 ts_tablet_manager.cc:1397] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:43:02.083135 29251 log.cc:824] T 7dbc56c7b3ad456fb758de777c488333 P 0729e58bc11546bdb97cced07a39f5dd: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:02.084115 29251 tablet_bootstrap.cc:492] T 7dbc56c7b3ad456fb758de777c488333 P 0729e58bc11546bdb97cced07a39f5dd: No bootstrap required, opened a new log
I20260620 18:43:02.084197 29251 ts_tablet_manager.cc:1397] T 7dbc56c7b3ad456fb758de777c488333 P 0729e58bc11546bdb97cced07a39f5dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:43:02.085073 29252 raft_consensus.cc:348] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729e58bc11546bdb97cced07a39f5dd" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36715 } } peers { permanent_uuid: "79b5a564434145deb09c4166b8761119" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35047 } } peers { permanent_uuid: "531634ef874543e885d6b5913f34d182" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33727 } }
I20260620 18:43:02.085271 29252 raft_consensus.cc:374] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:02.085320 29252 raft_consensus.cc:729] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 531634ef874543e885d6b5913f34d182, State: Initialized, Role: FOLLOWER
I20260620 18:43:02.085450 29252 consensus_queue.cc:260] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729e58bc11546bdb97cced07a39f5dd" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36715 } } peers { permanent_uuid: "79b5a564434145deb09c4166b8761119" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35047 } } peers { permanent_uuid: "531634ef874543e885d6b5913f34d182" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33727 } }
I20260620 18:43:02.085814 29250 raft_consensus.cc:348] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729e58bc11546bdb97cced07a39f5dd" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36715 } } peers { permanent_uuid: "79b5a564434145deb09c4166b8761119" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35047 } } peers { permanent_uuid: "531634ef874543e885d6b5913f34d182" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33727 } }
I20260620 18:43:02.085898 29236 heartbeater.cc:503] Master 127.10.250.126:42153 was elected leader, sending a full tablet report...
I20260620 18:43:02.085873 29252 ts_tablet_manager.cc:1428] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:43:02.086021 29250 raft_consensus.cc:374] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:02.086074 29250 raft_consensus.cc:729] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b5a564434145deb09c4166b8761119, State: Initialized, Role: FOLLOWER
I20260620 18:43:02.086308 29250 consensus_queue.cc:260] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729e58bc11546bdb97cced07a39f5dd" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36715 } } peers { permanent_uuid: "79b5a564434145deb09c4166b8761119" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35047 } } peers { permanent_uuid: "531634ef874543e885d6b5913f34d182" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33727 } }
I20260620 18:43:02.086709 29105 heartbeater.cc:503] Master 127.10.250.126:42153 was elected leader, sending a full tablet report...
I20260620 18:43:02.087443 29250 ts_tablet_manager.cc:1428] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260620 18:43:02.087844 29251 raft_consensus.cc:348] T 7dbc56c7b3ad456fb758de777c488333 P 0729e58bc11546bdb97cced07a39f5dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729e58bc11546bdb97cced07a39f5dd" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36715 } } peers { permanent_uuid: "79b5a564434145deb09c4166b8761119" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35047 } } peers { permanent_uuid: "531634ef874543e885d6b5913f34d182" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33727 } }
I20260620 18:43:02.088065 29251 raft_consensus.cc:374] T 7dbc56c7b3ad456fb758de777c488333 P 0729e58bc11546bdb97cced07a39f5dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:02.088126 29251 raft_consensus.cc:729] T 7dbc56c7b3ad456fb758de777c488333 P 0729e58bc11546bdb97cced07a39f5dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0729e58bc11546bdb97cced07a39f5dd, State: Initialized, Role: FOLLOWER
I20260620 18:43:02.088251 29251 consensus_queue.cc:260] T 7dbc56c7b3ad456fb758de777c488333 P 0729e58bc11546bdb97cced07a39f5dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729e58bc11546bdb97cced07a39f5dd" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36715 } } peers { permanent_uuid: "79b5a564434145deb09c4166b8761119" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35047 } } peers { permanent_uuid: "531634ef874543e885d6b5913f34d182" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33727 } }
I20260620 18:43:02.088500 29251 ts_tablet_manager.cc:1428] T 7dbc56c7b3ad456fb758de777c488333 P 0729e58bc11546bdb97cced07a39f5dd: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 18:43:02.088579 28974 heartbeater.cc:503] Master 127.10.250.126:42153 was elected leader, sending a full tablet report...
W20260620 18:43:02.155640 29106 tablet.cc:2369] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:43:02.261979 28975 tablet.cc:2369] T 7dbc56c7b3ad456fb758de777c488333 P 0729e58bc11546bdb97cced07a39f5dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:02.272123 29257 raft_consensus.cc:482] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:43:02.272336 29257 raft_consensus.cc:504] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729e58bc11546bdb97cced07a39f5dd" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36715 } } peers { permanent_uuid: "79b5a564434145deb09c4166b8761119" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35047 } } peers { permanent_uuid: "531634ef874543e885d6b5913f34d182" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33727 } }
I20260620 18:43:02.272750 29257 leader_election.cc:288] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0729e58bc11546bdb97cced07a39f5dd (127.10.250.65:36715), 531634ef874543e885d6b5913f34d182 (127.10.250.67:33727)
I20260620 18:43:02.276350 29247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.273137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:58941 (local address 127.10.250.65:36715)
0620 18:43:02.273225 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.273228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.273237 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:43:02.273274 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.273275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.273285 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.273365 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:02.273528 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.274048 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.275190 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.275692 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.276021 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.276035 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.276220 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.276270 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:43:02.276351 29107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.273003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:36715 (local address 127.10.250.66:58941)
0620 18:43:02.273075 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:02.273091 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:02.273165 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:02.273369 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:02.273370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:02.273375 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:02.273454 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.273456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.274150 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.274152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.275051 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.275054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.276011 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.276012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.276102 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.276106 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:43:02.276210 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:43:02.276263 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 18:43:02.276623 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.273003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:33727 (local address 127.10.250.66:58487)
0620 18:43:02.273326 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:02.273340 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:02.273452 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:02.273610 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:02.273611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:02.273612 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:02.273705 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.273707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.274428 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.274429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.275438 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.275441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.276298 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.276299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.276377 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.276380 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:43:02.276453 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 18:43:02.276503 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":95,"thread_start_us":97,"threads_started":1}
I20260620 18:43:02.276618 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.273155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:58487 (local address 127.10.250.67:33727)
0620 18:43:02.273203 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.273205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.273438 (+ 233us) server_negotiation.cc:407] Connection header received
0620 18:43:02.273531 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.273532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.273541 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.273607 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:02.273810 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.274347 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.275521 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.276164 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.276265 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.276285 (+ 20us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.276513 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.276557 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:43:02.276770 28928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbc56c7b3ad456fb758de777c488333" candidate_uuid: "79b5a564434145deb09c4166b8761119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0729e58bc11546bdb97cced07a39f5dd" is_pre_election: true
I20260620 18:43:02.276954 28928 raft_consensus.cc:2454] T 7dbc56c7b3ad456fb758de777c488333 P 0729e58bc11546bdb97cced07a39f5dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79b5a564434145deb09c4166b8761119 in term 0.
I20260620 18:43:02.276979 29190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbc56c7b3ad456fb758de777c488333" candidate_uuid: "79b5a564434145deb09c4166b8761119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "531634ef874543e885d6b5913f34d182" is_pre_election: true
I20260620 18:43:02.277149 29190 raft_consensus.cc:2454] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79b5a564434145deb09c4166b8761119 in term 0.
I20260620 18:43:02.277220 28995 leader_election.cc:302] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0729e58bc11546bdb97cced07a39f5dd, 79b5a564434145deb09c4166b8761119; no voters:
I20260620 18:43:02.277370 29257 raft_consensus.cc:2781] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:43:02.277441 29257 raft_consensus.cc:482] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:43:02.277468 29257 raft_consensus.cc:3037] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:02.278239 29257 raft_consensus.cc:504] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729e58bc11546bdb97cced07a39f5dd" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36715 } } peers { permanent_uuid: "79b5a564434145deb09c4166b8761119" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35047 } } peers { permanent_uuid: "531634ef874543e885d6b5913f34d182" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33727 } }
I20260620 18:43:02.278409 29257 leader_election.cc:288] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [CANDIDATE]: Term 1 election: Requested vote from peers 0729e58bc11546bdb97cced07a39f5dd (127.10.250.65:36715), 531634ef874543e885d6b5913f34d182 (127.10.250.67:33727)
I20260620 18:43:02.278596 29190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbc56c7b3ad456fb758de777c488333" candidate_uuid: "79b5a564434145deb09c4166b8761119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "531634ef874543e885d6b5913f34d182"
I20260620 18:43:02.278677 29190 raft_consensus.cc:3037] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:02.279356 29190 raft_consensus.cc:2454] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79b5a564434145deb09c4166b8761119 in term 1.
I20260620 18:43:02.279487 28928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbc56c7b3ad456fb758de777c488333" candidate_uuid: "79b5a564434145deb09c4166b8761119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0729e58bc11546bdb97cced07a39f5dd"
I20260620 18:43:02.279541 28994 leader_election.cc:302] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 531634ef874543e885d6b5913f34d182, 79b5a564434145deb09c4166b8761119; no voters:
I20260620 18:43:02.279677 29257 raft_consensus.cc:2781] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:02.279655 28928 raft_consensus.cc:3037] T 7dbc56c7b3ad456fb758de777c488333 P 0729e58bc11546bdb97cced07a39f5dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:02.279872 29257 raft_consensus.cc:686] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [term 1 LEADER]: Becoming Leader. State: Replica: 79b5a564434145deb09c4166b8761119, State: Running, Role: LEADER
I20260620 18:43:02.280000 29257 consensus_queue.cc:237] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729e58bc11546bdb97cced07a39f5dd" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36715 } } peers { permanent_uuid: "79b5a564434145deb09c4166b8761119" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35047 } } peers { permanent_uuid: "531634ef874543e885d6b5913f34d182" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33727 } }
I20260620 18:43:02.280750 28928 raft_consensus.cc:2454] T 7dbc56c7b3ad456fb758de777c488333 P 0729e58bc11546bdb97cced07a39f5dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79b5a564434145deb09c4166b8761119 in term 1.
I20260620 18:43:02.280989 28785 catalog_manager.cc:5495] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79b5a564434145deb09c4166b8761119 (127.10.250.66). New cstate: current_term: 1 leader_uuid: "79b5a564434145deb09c4166b8761119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729e58bc11546bdb97cced07a39f5dd" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79b5a564434145deb09c4166b8761119" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 35047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531634ef874543e885d6b5913f34d182" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33727 } health_report { overall_health: UNKNOWN } } }
W20260620 18:43:02.291059 29237 tablet.cc:2369] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:02.305517 29246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.302395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50614 (local address 127.10.250.66:35047)
0620 18:43:02.302475 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.302477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.302554 (+ 77us) server_negotiation.cc:407] Connection header received
0620 18:43:02.302698 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.302699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.302724 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.302822 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:02.303036 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.303808 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.304654 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.304888 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.304956 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.305110 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:02.305230 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:02.305312 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.305405 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.305427 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:43:02.307849 29190 raft_consensus.cc:1261] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182 [term 1 FOLLOWER]: Refusing update from remote peer 79b5a564434145deb09c4166b8761119: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:02.307850 28928 raft_consensus.cc:1261] T 7dbc56c7b3ad456fb758de777c488333 P 0729e58bc11546bdb97cced07a39f5dd [term 1 FOLLOWER]: Refusing update from remote peer 79b5a564434145deb09c4166b8761119: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:02.308297 29257 consensus_queue.cc:1035] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0729e58bc11546bdb97cced07a39f5dd" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 36715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:02.308521 29260 consensus_queue.cc:1035] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531634ef874543e885d6b5913f34d182" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 33727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:02.312306 29264 mvcc.cc:205] Tried to move back new op lower bound from 7298994103529680896 to 7298994103419498496. Current Snapshot: MvccSnapshot[applied={T|T < 7298994103529680896 or (T in {7298994103529680896})}]
I20260620 18:43:02.313741 28785 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:02.313861 28785 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:02.314770 28785 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cdd7b6f4885476288d2aff528bd1f8a: Sending DeleteTablet for 3 replicas of tablet 7dbc56c7b3ad456fb758de777c488333
I20260620 18:43:02.315208 28908 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbc56c7b3ad456fb758de777c488333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:02 UTC) from {username='slave'} at 127.0.0.1:40752
I20260620 18:43:02.315239 29170 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbc56c7b3ad456fb758de777c488333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:02 UTC) from {username='slave'} at 127.0.0.1:36882
I20260620 18:43:02.315665 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 28846
I20260620 18:43:02.315780 29039 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbc56c7b3ad456fb758de777c488333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:02 UTC) from {username='slave'} at 127.0.0.1:50604
I20260620 18:43:02.316706 29274 tablet_replica.cc:330] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119: stopping tablet replica
I20260620 18:43:02.316843 29274 raft_consensus.cc:2229] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:43:02.316955 29274 raft_consensus.cc:2258] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:02.317420 29274 ts_tablet_manager.cc:1905] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:02.319084 29274 ts_tablet_manager.cc:1918] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:02.319190 29274 log.cc:1196] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/7dbc56c7b3ad456fb758de777c488333
I20260620 18:43:02.319432 29274 ts_tablet_manager.cc:1939] T 7dbc56c7b3ad456fb758de777c488333 P 79b5a564434145deb09c4166b8761119: Deleting consensus metadata
I20260620 18:43:02.319897 28772 catalog_manager.cc:4841] TS 79b5a564434145deb09c4166b8761119 (127.10.250.66:35047): tablet 7dbc56c7b3ad456fb758de777c488333 (table test-table [id=14a2ec73de6f4ee38be6b0797b396d3d]) successfully deleted
I20260620 18:43:02.322686 29273 tablet_replica.cc:330] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182: stopping tablet replica
I20260620 18:43:02.322868 29273 raft_consensus.cc:2229] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:02.323035 29273 raft_consensus.cc:2258] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:02.323601 29273 ts_tablet_manager.cc:1905] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 18:43:02.323709 28773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:36715 (error 108)
I20260620 18:43:02.324296 29244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.324147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:36715 (local address 127.0.0.1:40766)
0620 18:43:02.324206 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:02.324253 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:36715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 18:43:02.324406 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 28977
W20260620 18:43:02.324599 28773 catalog_manager.cc:4584] TS 0729e58bc11546bdb97cced07a39f5dd (127.10.250.65:36715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7dbc56c7b3ad456fb758de777c488333: Network error: Client connection negotiation failed: client connection to 127.10.250.65:36715: connect: Connection refused (error 111)
I20260620 18:43:02.325775 29273 ts_tablet_manager.cc:1918] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:02.326011 29273 log.cc:1196] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/7dbc56c7b3ad456fb758de777c488333
I20260620 18:43:02.326267 29273 ts_tablet_manager.cc:1939] T 7dbc56c7b3ad456fb758de777c488333 P 531634ef874543e885d6b5913f34d182: Deleting consensus metadata
I20260620 18:43:02.326843 28772 catalog_manager.cc:4841] TS 531634ef874543e885d6b5913f34d182 (127.10.250.67:33727): tablet 7dbc56c7b3ad456fb758de777c488333 (table test-table [id=14a2ec73de6f4ee38be6b0797b396d3d]) successfully deleted
I20260620 18:43:02.330039 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 29108
I20260620 18:43:02.335907 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 28755
2026-06-20T18:43:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (895 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:02 dist-test-slave-wp6g krb5kdc[29278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:02 dist-test-slave-wp6g krb5kdc[29278](info): set up 2 sockets
Jun 20 18:43:02 dist-test-slave-wp6g krb5kdc[29278](info): commencing operation
krb5kdc: starting...
W20260620 18:43:03.402375 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:03 dist-test-slave-wp6g krb5kdc[29278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:43:03.529606 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:39683
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:35353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:39683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:03.613750 29291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:03.614030 29291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:03.614073 29291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:03.615368 29291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:03.615458 29291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:03.615481 29291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:03.615501 29291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:03.615528 29291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:03.617113 29291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:35353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:39683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:39683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:03.617410 29291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:03.617662 29291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:03.622813 29296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:03.622821 29299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:03.622821 29297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:03.622869 29291 server_base.cc:1034] running on GCE node
I20260620 18:43:03.623294 29291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:03.623560 29291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:03.624737 29291 hybrid_clock.cc:648] HybridClock initialized: now 1781980983624717 us; error 31 us; skew 500 ppm
Jun 20 18:43:03 dist-test-slave-wp6g krb5kdc[29278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980983, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:03.627527 29291 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:43:03.628085 29291 webserver.cc:460] Webserver started at http://127.10.250.126:37397/ using document root <none> and password file <none>
I20260620 18:43:03.628338 29291 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:03.628408 29291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:03.628566 29291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:03.629703 29291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "d0ff4321a82148c4b09e4c91fefb4d9e"
format_stamp: "Formatted at 2026-06-20 18:43:03 on dist-test-slave-wp6g"
server_key: "eeec98ff4f37e8bc28e5d9d6666c72a7"
server_key_iv: "567aa518a8f7b6b93f7834c36a6ce7a9"
server_key_version: "encryptionkey@0"
I20260620 18:43:03.630080 29291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "d0ff4321a82148c4b09e4c91fefb4d9e"
format_stamp: "Formatted at 2026-06-20 18:43:03 on dist-test-slave-wp6g"
server_key: "eeec98ff4f37e8bc28e5d9d6666c72a7"
server_key_iv: "567aa518a8f7b6b93f7834c36a6ce7a9"
server_key_version: "encryptionkey@0"
I20260620 18:43:03.631695 29291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:43:03.632651 29306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:03.632916 29291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:03.633035 29291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "d0ff4321a82148c4b09e4c91fefb4d9e"
format_stamp: "Formatted at 2026-06-20 18:43:03 on dist-test-slave-wp6g"
server_key: "eeec98ff4f37e8bc28e5d9d6666c72a7"
server_key_iv: "567aa518a8f7b6b93f7834c36a6ce7a9"
server_key_version: "encryptionkey@0"
I20260620 18:43:03.633126 29291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:03.650095 29291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:03.652529 29291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:03.652776 29291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:03.657337 29291 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:39683
I20260620 18:43:03.657380 29358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:39683 every 8 connection(s)
I20260620 18:43:03.657755 29291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:03.658707 29359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:03.661229 29359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e: Bootstrap starting.
I20260620 18:43:03.661873 29359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:03.662200 29359 log.cc:824] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:03.663146 29359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e: No bootstrap required, opened a new log
I20260620 18:43:03.664839 29359 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ff4321a82148c4b09e4c91fefb4d9e" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39683 } }
I20260620 18:43:03.664984 29359 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:03.665007 29359 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0ff4321a82148c4b09e4c91fefb4d9e, State: Initialized, Role: FOLLOWER
I20260620 18:43:03.665194 29359 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ff4321a82148c4b09e4c91fefb4d9e" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39683 } }
I20260620 18:43:03.665289 29359 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:03.665334 29359 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:03.665393 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 29291
I20260620 18:43:03.665409 29359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:03.665529 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:03.665753 11241 external_mini_cluster.cc:1428] Setting key c4c6b2d5651dc29602cff3fc4c46588d
I20260620 18:43:03.666152 29359 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ff4321a82148c4b09e4c91fefb4d9e" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39683 } }
I20260620 18:43:03.666271 29359 leader_election.cc:302] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0ff4321a82148c4b09e4c91fefb4d9e; no voters:
I20260620 18:43:03.666496 29359 leader_election.cc:288] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:03.666702 29365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:03.666826 29359 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:03.666844 29365 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [term 1 LEADER]: Becoming Leader. State: Replica: d0ff4321a82148c4b09e4c91fefb4d9e, State: Running, Role: LEADER
I20260620 18:43:03.666929 29365 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ff4321a82148c4b09e4c91fefb4d9e" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39683 } }
I20260620 18:43:03.667268 29365 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0ff4321a82148c4b09e4c91fefb4d9e. Latest consensus state: current_term: 1 leader_uuid: "d0ff4321a82148c4b09e4c91fefb4d9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ff4321a82148c4b09e4c91fefb4d9e" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39683 } } }
I20260620 18:43:03.667372 29365 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:03.667590 29366 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0ff4321a82148c4b09e4c91fefb4d9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ff4321a82148c4b09e4c91fefb4d9e" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39683 } } }
I20260620 18:43:03.667776 29366 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:03.668033 29372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:03.668808 29372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:03.670617 29372 catalog_manager.cc:1347] Generated new cluster ID: 09f8b697e1764d518540a2653e2ee584
I20260620 18:43:03.670681 29372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:43:03 dist-test-slave-wp6g krb5kdc[29278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:03.677613 29372 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:03.678416 29372 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:03.679018 29363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:03.666325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.10.250.126:39683)
0620 18:43:03.666589 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:03.666601 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:03.666652 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:43:03.666851 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:03.666860 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:03.666933 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:03.668062 (+ 1129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:03.668375 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.669567 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.670817 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.671529 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.671659 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:03.674756 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:03.674762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:03.674762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:03.674776 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:03.676992 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:03.677527 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:03.677531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:03.677532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:03.677662 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:03.677899 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:03.677900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:03.677900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:03.678248 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:03.678411 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:03.678639 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 18:43:03.678793 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":168,"thread_start_us":133,"threads_started":1}
I20260620 18:43:03.688282 29372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e: Generated new TSK 0
I20260620 18:43:03.688541 29372 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:43:03.719576 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:39683
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:35353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:03.814169 29385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:03.814432 29385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:03.814476 29385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:03.815729 29385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:03.815797 29385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:03.815845 29385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:43:03.817466 29385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:35353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:39683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:03.817750 29385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:03.817998 29385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:03.823746 29390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:03.823745 29393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:03.823751 29391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:03.824020 29385 server_base.cc:1034] running on GCE node
I20260620 18:43:03.824254 29385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:03.824554 29385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:03.825755 29385 hybrid_clock.cc:648] HybridClock initialized: now 1781980983825742 us; error 41 us; skew 500 ppm
Jun 20 18:43:03 dist-test-slave-wp6g krb5kdc[29278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980983, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:03.828500 29385 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:43:03.828930 29385 webserver.cc:460] Webserver started at http://127.10.250.65:43423/ using document root <none> and password file <none>
I20260620 18:43:03.829171 29385 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:03.829224 29385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:03.829368 29385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:03.830888 29385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "e7749da5db6f46a2a880a0a91caa976a"
format_stamp: "Formatted at 2026-06-20 18:43:03 on dist-test-slave-wp6g"
server_key: "45d894c4b10f14019304d95680c8c47c"
server_key_iv: "60354f35a59a01502b23c358a34819ee"
server_key_version: "encryptionkey@0"
I20260620 18:43:03.831319 29385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "e7749da5db6f46a2a880a0a91caa976a"
format_stamp: "Formatted at 2026-06-20 18:43:03 on dist-test-slave-wp6g"
server_key: "45d894c4b10f14019304d95680c8c47c"
server_key_iv: "60354f35a59a01502b23c358a34819ee"
server_key_version: "encryptionkey@0"
I20260620 18:43:03.832803 29385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:43:03.833706 29400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:03.833894 29385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:03.833993 29385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "e7749da5db6f46a2a880a0a91caa976a"
format_stamp: "Formatted at 2026-06-20 18:43:03 on dist-test-slave-wp6g"
server_key: "45d894c4b10f14019304d95680c8c47c"
server_key_iv: "60354f35a59a01502b23c358a34819ee"
server_key_version: "encryptionkey@0"
I20260620 18:43:03.834071 29385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:03.847579 29385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:03.850258 29385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:03.850411 29385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:03.850713 29385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:03.851033 29385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:03.851068 29385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:03.851100 29385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:03.851114 29385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:03.857640 29385 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:43001
I20260620 18:43:03.857712 29513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:43001 every 8 connection(s)
I20260620 18:43:03.858064 29385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:43:03 dist-test-slave-wp6g krb5kdc[29278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980983, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:03.865689 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 29385
I20260620 18:43:03.865806 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:43:03.866000 11241 external_mini_cluster.cc:1428] Setting key 6ff2beee9b253e2bb92ef37caae2ee56
I20260620 18:43:03.868898 29363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:03.859182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:60271 (local address 127.10.250.126:39683)
0620 18:43:03.859257 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:03.859259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:03.859341 (+ 82us) server_negotiation.cc:407] Connection header received
0620 18:43:03.859890 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:03.859892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:03.859902 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:03.859972 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:03.861151 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.861818 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.862957 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.863174 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.863237 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:03.865569 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:03.865570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:03.865571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:03.865580 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:03.867419 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:03.867868 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:03.867869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:03.867870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:03.867935 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:03.868214 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:03.868214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:03.868215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:03.868345 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:03.868425 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:03.868712 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 18:43:03.868805 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:43:03.869118 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:03.859009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:39683 (local address 127.10.250.65:60271)
0620 18:43:03.859191 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:03.859238 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:03.859762 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:03.860088 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:03.860092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:03.860765 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:03.860981 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:03.860987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.861933 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.861934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:03.862873 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:03.862876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.863285 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.863286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:03.863403 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:03.863929 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:03.863938 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:03.865463 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:03.867502 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:03.867512 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:03.867514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:03.867773 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:03.868012 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:03.868012 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:03.868013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:03.868133 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:03.868414 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:03.868421 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:03.868573 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 18:43:03.868689 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:43:03.868864 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20260620 18:43:03.869581 29514 heartbeater.cc:347] Connected to a master server at 127.10.250.126:39683
I20260620 18:43:03.869676 29514 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:03.869882 29514 heartbeater.cc:511] Master 127.10.250.126:39683 requested a full tablet report, sending...
I20260620 18:43:03.870445 29323 ts_manager.cc:194] Registered new tserver with Master: e7749da5db6f46a2a880a0a91caa976a (127.10.250.65:43001)
I20260620 18:43:03.871121 29323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:60271
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:43:03.895565 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:39683
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:35353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:03.979583 29519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:03.979815 29519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:03.979840 29519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:03.980947 29519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:03.980983 29519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:03.981009 29519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:43:03.982419 29519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:35353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:39683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:03.982641 29519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:03.982821 29519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:03.988307 29527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:03.988359 29525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:03.988360 29524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:03.988444 29519 server_base.cc:1034] running on GCE node
I20260620 18:43:03.988754 29519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:03.989022 29519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:03.990202 29519 hybrid_clock.cc:648] HybridClock initialized: now 1781980983990179 us; error 39 us; skew 500 ppm
Jun 20 18:43:03 dist-test-slave-wp6g krb5kdc[29278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980983, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:03.993022 29519 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:43:03.993508 29519 webserver.cc:460] Webserver started at http://127.10.250.66:43151/ using document root <none> and password file <none>
I20260620 18:43:03.993714 29519 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:03.993758 29519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:03.993851 29519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:03.994982 29519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "4da87a745a9b4740bb790e2f7e02a4b2"
format_stamp: "Formatted at 2026-06-20 18:43:03 on dist-test-slave-wp6g"
server_key: "9343fb146abfc21f913f8380ad7d2739"
server_key_iv: "e0144b094fbbdb50f00c01703dcc2c41"
server_key_version: "encryptionkey@0"
I20260620 18:43:03.995304 29519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "4da87a745a9b4740bb790e2f7e02a4b2"
format_stamp: "Formatted at 2026-06-20 18:43:03 on dist-test-slave-wp6g"
server_key: "9343fb146abfc21f913f8380ad7d2739"
server_key_iv: "e0144b094fbbdb50f00c01703dcc2c41"
server_key_version: "encryptionkey@0"
I20260620 18:43:03.996721 29519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:43:03.997680 29534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:03.998009 29519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:03.998112 29519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "4da87a745a9b4740bb790e2f7e02a4b2"
format_stamp: "Formatted at 2026-06-20 18:43:03 on dist-test-slave-wp6g"
server_key: "9343fb146abfc21f913f8380ad7d2739"
server_key_iv: "e0144b094fbbdb50f00c01703dcc2c41"
server_key_version: "encryptionkey@0"
I20260620 18:43:03.998188 29519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:04.017082 29519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:04.019497 29519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:04.019671 29519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:04.019949 29519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:04.020287 29519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:04.020323 29519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:04.020344 29519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:04.020357 29519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:04.026561 29519 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:37501
I20260620 18:43:04.026608 29647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:37501 every 8 connection(s)
I20260620 18:43:04.026981 29519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:43:04.031064 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 29519
I20260620 18:43:04.031217 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:43:04.031426 11241 external_mini_cluster.cc:1428] Setting key b969d13e4095e835bb15a9aa87570d13
Jun 20 18:43:04 dist-test-slave-wp6g krb5kdc[29278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980983, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:04.037997 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.027782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:39683 (local address 127.10.250.66:51105)
0620 18:43:04.027992 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:04.028042 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:04.028593 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:04.028890 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:04.028896 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:04.029541 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:04.029751 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.029756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.030478 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.030479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.031453 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.031455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.031899 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.031900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.032037 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.032697 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:04.032708 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:04.034472 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:04.036394 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:04.036403 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:04.036406 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:04.036672 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:04.036910 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:04.036911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:04.036911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:04.037030 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:04.037428 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:04.037435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:04.037585 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 18:43:04.037680 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:43:04.037818 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
I20260620 18:43:04.037993 29363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.027811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:51105 (local address 127.10.250.126:39683)
0620 18:43:04.027933 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:04.027935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:04.028135 (+ 200us) server_negotiation.cc:407] Connection header received
0620 18:43:04.028737 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:04.028738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:04.028750 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:04.028810 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:04.029851 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.030369 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.031568 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.031809 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.031870 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.034696 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:04.034698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:04.034698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:04.034706 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:04.036307 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:04.036758 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:04.036759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:04.036760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:04.036836 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:04.037140 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:04.037141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:04.037141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:04.037322 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:04.037433 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:04.037731 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 18:43:04.037854 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:43:04.038568 29648 heartbeater.cc:347] Connected to a master server at 127.10.250.126:39683
I20260620 18:43:04.038697 29648 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:04.038910 29648 heartbeater.cc:511] Master 127.10.250.126:39683 requested a full tablet report, sending...
I20260620 18:43:04.039330 29323 ts_manager.cc:194] Registered new tserver with Master: 4da87a745a9b4740bb790e2f7e02a4b2 (127.10.250.66:37501)
I20260620 18:43:04.039727 29323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:51105
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:43:04.062157 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:39683
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:35353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:04.145707 29653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:04.145937 29653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:04.145965 29653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:04.147460 29653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:04.147516 29653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:04.147547 29653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:43:04.149123 29653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:35353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:39683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:04.149415 29653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:04.149678 29653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:04.154997 29658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:04.155004 29659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:04.154997 29661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:04.155090 29653 server_base.cc:1034] running on GCE node
I20260620 18:43:04.155381 29653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:04.155652 29653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:04.156847 29653 hybrid_clock.cc:648] HybridClock initialized: now 1781980984156824 us; error 39 us; skew 500 ppm
Jun 20 18:43:04 dist-test-slave-wp6g krb5kdc[29278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980984, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:04.159507 29653 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:43:04.160012 29653 webserver.cc:460] Webserver started at http://127.10.250.67:33411/ using document root <none> and password file <none>
I20260620 18:43:04.160274 29653 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:04.160346 29653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:04.160497 29653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:04.161964 29653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "487d44949ac84e0f97504d2965739938"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-wp6g"
server_key: "b830234cb8898c744ada24c22d4f97bc"
server_key_iv: "1045cf5c0ac79828be624149d96c9f02"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.162406 29653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "487d44949ac84e0f97504d2965739938"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-wp6g"
server_key: "b830234cb8898c744ada24c22d4f97bc"
server_key_iv: "1045cf5c0ac79828be624149d96c9f02"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.163849 29653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:04.164674 29668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:04.164880 29653 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:04.164983 29653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "487d44949ac84e0f97504d2965739938"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-wp6g"
server_key: "b830234cb8898c744ada24c22d4f97bc"
server_key_iv: "1045cf5c0ac79828be624149d96c9f02"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.165082 29653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:04.175470 29653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:04.178087 29653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:04.178310 29653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:04.178671 29653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:04.179051 29653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:04.179113 29653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:04.179162 29653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:04.179193 29653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:04.185757 29653 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:36009
I20260620 18:43:04.185802 29781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:36009 every 8 connection(s)
I20260620 18:43:04.186170 29653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:43:04.187479 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 29653
I20260620 18:43:04.187569 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:43:04.187772 11241 external_mini_cluster.cc:1428] Setting key 921a096692a3a65e60f00ee80765bd96
Jun 20 18:43:04 dist-test-slave-wp6g krb5kdc[29278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980984, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:04.196832 29363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.186997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:33759 (local address 127.10.250.126:39683)
0620 18:43:04.187068 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:04.187070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:04.187353 (+ 283us) server_negotiation.cc:407] Connection header received
0620 18:43:04.187950 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:04.187952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:04.187992 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:04.188097 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:04.189365 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.189923 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.191119 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.191331 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.191398 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.193864 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:04.193866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:04.193866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:04.193875 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:04.195290 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:04.195755 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:04.195756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:04.195757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:04.195817 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:04.196104 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:04.196104 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:04.196105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:04.196226 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:04.196299 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:04.196645 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 18:43:04.196733 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:43:04.197046 29784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.186979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:39683 (local address 127.10.250.67:33759)
0620 18:43:04.187161 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:04.187215 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:04.187834 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:04.188268 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:04.188272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:04.189061 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:04.189272 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.189278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.190017 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.190018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.190990 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.190992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.191415 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.191416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.191533 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.192153 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:04.192162 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:04.193729 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:04.195396 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:04.195404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:04.195406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:04.195659 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:04.195908 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:04.195909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:04.195910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:04.196026 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:04.196300 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:04.196308 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:04.196457 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 18:43:04.196611 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 18:43:04.196794 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":108,"threads_started":1}
I20260620 18:43:04.197515 29782 heartbeater.cc:347] Connected to a master server at 127.10.250.126:39683
I20260620 18:43:04.197602 29782 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:04.197748 29782 heartbeater.cc:511] Master 127.10.250.126:39683 requested a full tablet report, sending...
I20260620 18:43:04.198052 29323 ts_manager.cc:194] Registered new tserver with Master: 487d44949ac84e0f97504d2965739938 (127.10.250.67:36009)
I20260620 18:43:04.198457 29323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:33759
I20260620 18:43:04.199772 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:43:04.208737 29363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.201104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.10.250.126:39683)
0620 18:43:04.201169 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:04.201171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:04.201330 (+ 159us) server_negotiation.cc:407] Connection header received
0620 18:43:04.201422 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:04.201423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:04.201430 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:04.201483 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:04.202043 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.202700 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.203422 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.203612 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.203672 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.205354 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:04.205355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:04.205356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:04.205362 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:04.207104 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:04.207557 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:04.207558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:04.207559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:04.207620 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:04.208096 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:04.208096 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:04.208097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:04.208262 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:04.208362 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:04.208605 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 18:43:04.208667 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:43:04.210853 29323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:43:04.212451 29323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:43:04.219223 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.214990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:37501 (local address 127.0.0.1:35140)
0620 18:43:04.215276 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:04.215290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:04.215346 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:04.215624 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:04.215627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:04.215630 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:04.215726 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.215729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.216841 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.216842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.218107 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.218110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.218911 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.218912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.218990 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.219005 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 18:43:04.219081 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 18:43:04.219138 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":83,"threads_started":1}
I20260620 18:43:04.219233 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.215083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.10.250.66:37501)
0620 18:43:04.215451 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:04.215457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:04.215468 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:43:04.215512 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:04.215514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:04.215530 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:04.215613 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:04.215975 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.216739 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.218215 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.218823 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.218912 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.218974 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:04.219096 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 18:43:04.219138 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":80,"threads_started":1}
I20260620 18:43:04.219390 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.214988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:43001 (local address 127.0.0.1:53654)
0620 18:43:04.215163 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:04.215208 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:04.215278 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:04.215770 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:04.215771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:04.215774 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:04.215812 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.215813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.217107 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.217107 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.218231 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.218233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.219143 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.219143 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.219198 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.219202 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:43:04.219231 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 18:43:04.219261 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":30,"thread_start_us":48,"threads_started":1}
I20260620 18:43:04.219489 29798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.215406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.10.250.67:36009)
0620 18:43:04.215596 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:04.215601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:04.215611 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:43:04.215658 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:04.215660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:04.215676 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:04.215758 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:04.215966 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.217023 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.218155 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.218928 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.218999 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.219057 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:04.219349 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 18:43:04.219391 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":98,"threads_started":1}
I20260620 18:43:04.219576 29797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.215180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:36009 (local address 127.0.0.1:58096)
0620 18:43:04.215387 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:04.215406 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:04.215443 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:04.215835 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:04.215836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:04.215838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:04.215888 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.215890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.217154 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.217155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.218070 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.218073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.219094 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.219095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.219165 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.219170 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:43:04.219217 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 18:43:04.219260 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":18,"thread_start_us":62,"threads_started":1}
I20260620 18:43:04.220089 29796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.215080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53654 (local address 127.10.250.65:43001)
0620 18:43:04.215632 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:04.215636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:04.215645 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:43:04.215683 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:04.215685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:04.215700 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:04.215779 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:04.215859 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.216995 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.218312 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.219042 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.219663 (+ 621us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.219854 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:04.219880 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 18:43:04.219969 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":97,"threads_started":1}
I20260620 18:43:04.220419 29716 tablet_service.cc:1467] Processing CreateTablet for tablet f3899b3056a9459f93eb6abb06b4cffb (DEFAULT_TABLE table=test-table [id=036af33cc4d54a838cc2d82b78f328ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:04.220536 29582 tablet_service.cc:1467] Processing CreateTablet for tablet f3899b3056a9459f93eb6abb06b4cffb (DEFAULT_TABLE table=test-table [id=036af33cc4d54a838cc2d82b78f328ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:04.220718 29448 tablet_service.cc:1467] Processing CreateTablet for tablet f3899b3056a9459f93eb6abb06b4cffb (DEFAULT_TABLE table=test-table [id=036af33cc4d54a838cc2d82b78f328ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:04.220914 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3899b3056a9459f93eb6abb06b4cffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:04.220917 29716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3899b3056a9459f93eb6abb06b4cffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:04.221015 29448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3899b3056a9459f93eb6abb06b4cffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:04.223846 29799 tablet_bootstrap.cc:492] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938: Bootstrap starting.
I20260620 18:43:04.224328 29800 tablet_bootstrap.cc:492] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a: Bootstrap starting.
I20260620 18:43:04.224774 29799 tablet_bootstrap.cc:654] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:04.225052 29800 tablet_bootstrap.cc:654] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:04.225420 29799 log.cc:824] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:04.225663 29800 log.cc:824] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:04.226085 29801 tablet_bootstrap.cc:492] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2: Bootstrap starting.
I20260620 18:43:04.226357 29799 tablet_bootstrap.cc:492] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938: No bootstrap required, opened a new log
I20260620 18:43:04.226441 29799 ts_tablet_manager.cc:1397] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:43:04.226459 29800 tablet_bootstrap.cc:492] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a: No bootstrap required, opened a new log
I20260620 18:43:04.226547 29800 ts_tablet_manager.cc:1397] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:43:04.226801 29801 tablet_bootstrap.cc:654] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:04.227386 29801 log.cc:824] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:04.228228 29801 tablet_bootstrap.cc:492] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2: No bootstrap required, opened a new log
I20260620 18:43:04.228308 29801 ts_tablet_manager.cc:1397] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:43:04.228349 29800 raft_consensus.cc:348] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7749da5db6f46a2a880a0a91caa976a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43001 } } peers { permanent_uuid: "487d44949ac84e0f97504d2965739938" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36009 } } peers { permanent_uuid: "4da87a745a9b4740bb790e2f7e02a4b2" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37501 } }
I20260620 18:43:04.228549 29800 raft_consensus.cc:374] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:04.228591 29800 raft_consensus.cc:729] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7749da5db6f46a2a880a0a91caa976a, State: Initialized, Role: FOLLOWER
I20260620 18:43:04.228725 29800 consensus_queue.cc:260] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7749da5db6f46a2a880a0a91caa976a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43001 } } peers { permanent_uuid: "487d44949ac84e0f97504d2965739938" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36009 } } peers { permanent_uuid: "4da87a745a9b4740bb790e2f7e02a4b2" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37501 } }
I20260620 18:43:04.229022 29800 ts_tablet_manager.cc:1428] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:43:04.229169 29514 heartbeater.cc:503] Master 127.10.250.126:39683 was elected leader, sending a full tablet report...
I20260620 18:43:04.229082 29799 raft_consensus.cc:348] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7749da5db6f46a2a880a0a91caa976a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43001 } } peers { permanent_uuid: "487d44949ac84e0f97504d2965739938" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36009 } } peers { permanent_uuid: "4da87a745a9b4740bb790e2f7e02a4b2" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37501 } }
I20260620 18:43:04.229259 29799 raft_consensus.cc:374] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:04.229295 29799 raft_consensus.cc:729] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 487d44949ac84e0f97504d2965739938, State: Initialized, Role: FOLLOWER
I20260620 18:43:04.229426 29799 consensus_queue.cc:260] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7749da5db6f46a2a880a0a91caa976a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43001 } } peers { permanent_uuid: "487d44949ac84e0f97504d2965739938" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36009 } } peers { permanent_uuid: "4da87a745a9b4740bb790e2f7e02a4b2" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37501 } }
I20260620 18:43:04.229687 29799 ts_tablet_manager.cc:1428] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 18:43:04.229811 29782 heartbeater.cc:503] Master 127.10.250.126:39683 was elected leader, sending a full tablet report...
I20260620 18:43:04.230577 29801 raft_consensus.cc:348] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7749da5db6f46a2a880a0a91caa976a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43001 } } peers { permanent_uuid: "487d44949ac84e0f97504d2965739938" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36009 } } peers { permanent_uuid: "4da87a745a9b4740bb790e2f7e02a4b2" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37501 } }
I20260620 18:43:04.230791 29801 raft_consensus.cc:374] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:04.230840 29801 raft_consensus.cc:729] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da87a745a9b4740bb790e2f7e02a4b2, State: Initialized, Role: FOLLOWER
I20260620 18:43:04.230953 29801 consensus_queue.cc:260] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7749da5db6f46a2a880a0a91caa976a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43001 } } peers { permanent_uuid: "487d44949ac84e0f97504d2965739938" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36009 } } peers { permanent_uuid: "4da87a745a9b4740bb790e2f7e02a4b2" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37501 } }
I20260620 18:43:04.231244 29801 ts_tablet_manager.cc:1428] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:43:04.231268 29648 heartbeater.cc:503] Master 127.10.250.126:39683 was elected leader, sending a full tablet report...
W20260620 18:43:04.277570 29649 tablet.cc:2369] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:43:04.358982 29515 tablet.cc:2369] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:04.430986 29807 raft_consensus.cc:482] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:43:04.431143 29807 raft_consensus.cc:504] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7749da5db6f46a2a880a0a91caa976a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43001 } } peers { permanent_uuid: "487d44949ac84e0f97504d2965739938" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36009 } } peers { permanent_uuid: "4da87a745a9b4740bb790e2f7e02a4b2" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37501 } }
I20260620 18:43:04.431631 29807 leader_election.cc:288] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7749da5db6f46a2a880a0a91caa976a (127.10.250.65:43001), 487d44949ac84e0f97504d2965739938 (127.10.250.67:36009)
I20260620 18:43:04.435465 29796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.432149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:60881 (local address 127.10.250.65:43001)
0620 18:43:04.432208 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:04.432210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:04.432221 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:43:04.432320 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:04.432321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:04.432332 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:04.432394 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:04.432650 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.433241 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.434577 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.435022 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.435081 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.435089 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:04.435358 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 18:43:04.435406 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 18:43:04.435461 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.432013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:43001 (local address 127.10.250.66:60881)
0620 18:43:04.432130 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:04.432146 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:04.432236 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:04.432429 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:04.432430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:04.432433 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:04.432529 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.432536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.433432 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.433433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.434348 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.434351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.435116 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.435117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.435242 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.435246 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:43:04.435320 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 18:43:04.435370 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 18:43:04.435971 29798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.432336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:45387 (local address 127.10.250.67:36009)
0620 18:43:04.432420 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:04.432422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:04.432431 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:43:04.432513 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:04.432514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:04.432524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:04.432596 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:04.432867 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.433585 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.434831 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.435552 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.435625 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.435635 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:04.435855 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 18:43:04.435901 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:43:04.435949 29468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3899b3056a9459f93eb6abb06b4cffb" candidate_uuid: "4da87a745a9b4740bb790e2f7e02a4b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7749da5db6f46a2a880a0a91caa976a" is_pre_election: true
I20260620 18:43:04.436193 29468 raft_consensus.cc:2454] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da87a745a9b4740bb790e2f7e02a4b2 in term 0.
I20260620 18:43:04.436211 29808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.432173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:36009 (local address 127.10.250.66:45387)
0620 18:43:04.432378 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:04.432389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:04.432437 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:04.432651 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:04.432652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:04.432654 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:04.432731 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.432733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.433718 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.433719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.434665 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.434676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.435697 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.435698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.435773 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.435777 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:43:04.436105 (+ 328us) client_negotiation.cc:240] Negotiation successful
0620 18:43:04.436156 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":128,"threads_started":1}
I20260620 18:43:04.436460 29537 leader_election.cc:302] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da87a745a9b4740bb790e2f7e02a4b2, e7749da5db6f46a2a880a0a91caa976a; no voters:
I20260620 18:43:04.436617 29807 raft_consensus.cc:2781] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:43:04.436750 29807 raft_consensus.cc:482] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:43:04.436780 29807 raft_consensus.cc:3037] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [term 0 FOLLOWER]: Advancing to term 1
W20260620 18:43:04.436834 29783 tablet.cc:2369] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:04.436812 29736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3899b3056a9459f93eb6abb06b4cffb" candidate_uuid: "4da87a745a9b4740bb790e2f7e02a4b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "487d44949ac84e0f97504d2965739938" is_pre_election: true
I20260620 18:43:04.436976 29736 raft_consensus.cc:2454] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da87a745a9b4740bb790e2f7e02a4b2 in term 0.
I20260620 18:43:04.437927 29807 raft_consensus.cc:504] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7749da5db6f46a2a880a0a91caa976a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43001 } } peers { permanent_uuid: "487d44949ac84e0f97504d2965739938" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36009 } } peers { permanent_uuid: "4da87a745a9b4740bb790e2f7e02a4b2" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37501 } }
I20260620 18:43:04.438196 29807 leader_election.cc:288] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [CANDIDATE]: Term 1 election: Requested vote from peers e7749da5db6f46a2a880a0a91caa976a (127.10.250.65:43001), 487d44949ac84e0f97504d2965739938 (127.10.250.67:36009)
I20260620 18:43:04.438395 29468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3899b3056a9459f93eb6abb06b4cffb" candidate_uuid: "4da87a745a9b4740bb790e2f7e02a4b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7749da5db6f46a2a880a0a91caa976a"
I20260620 18:43:04.438418 29736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3899b3056a9459f93eb6abb06b4cffb" candidate_uuid: "4da87a745a9b4740bb790e2f7e02a4b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "487d44949ac84e0f97504d2965739938"
I20260620 18:43:04.438501 29736 raft_consensus.cc:3037] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:04.438503 29468 raft_consensus.cc:3037] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:04.439473 29736 raft_consensus.cc:2454] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da87a745a9b4740bb790e2f7e02a4b2 in term 1.
I20260620 18:43:04.439473 29468 raft_consensus.cc:2454] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da87a745a9b4740bb790e2f7e02a4b2 in term 1.
I20260620 18:43:04.439700 29537 leader_election.cc:302] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da87a745a9b4740bb790e2f7e02a4b2, e7749da5db6f46a2a880a0a91caa976a; no voters:
I20260620 18:43:04.439833 29807 raft_consensus.cc:2781] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:04.440030 29807 raft_consensus.cc:686] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4da87a745a9b4740bb790e2f7e02a4b2, State: Running, Role: LEADER
I20260620 18:43:04.440155 29807 consensus_queue.cc:237] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7749da5db6f46a2a880a0a91caa976a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43001 } } peers { permanent_uuid: "487d44949ac84e0f97504d2965739938" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36009 } } peers { permanent_uuid: "4da87a745a9b4740bb790e2f7e02a4b2" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37501 } }
I20260620 18:43:04.441262 29322 catalog_manager.cc:5495] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da87a745a9b4740bb790e2f7e02a4b2 (127.10.250.66). New cstate: current_term: 1 leader_uuid: "4da87a745a9b4740bb790e2f7e02a4b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7749da5db6f46a2a880a0a91caa976a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "487d44949ac84e0f97504d2965739938" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da87a745a9b4740bb790e2f7e02a4b2" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 37501 } health_report { overall_health: HEALTHY } } }
I20260620 18:43:04.448137 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.445522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35144 (local address 127.10.250.66:37501)
0620 18:43:04.445584 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:04.445586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:04.445594 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:43:04.445638 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:04.445639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:04.445656 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:04.445739 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:04.445900 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.446395 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.447284 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.447502 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.447582 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.447746 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:04.447852 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:04.447970 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:04.448001 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 18:43:04.448014 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:43:04.450374 29736 raft_consensus.cc:1261] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938 [term 1 FOLLOWER]: Refusing update from remote peer 4da87a745a9b4740bb790e2f7e02a4b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:04.450475 29468 raft_consensus.cc:1261] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a [term 1 FOLLOWER]: Refusing update from remote peer 4da87a745a9b4740bb790e2f7e02a4b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:04.450891 29809 consensus_queue.cc:1035] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "487d44949ac84e0f97504d2965739938" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 36009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:04.451054 29807 consensus_queue.cc:1035] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7749da5db6f46a2a880a0a91caa976a" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 43001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:04.453699 29812 mvcc.cc:205] Tried to move back new op lower bound from 7298994112305311744 to 7298994112267624448. Current Snapshot: MvccSnapshot[applied={T|T < 7298994112305311744}]
I20260620 18:43:04.454001 29813 mvcc.cc:205] Tried to move back new op lower bound from 7298994112305311744 to 7298994112267624448. Current Snapshot: MvccSnapshot[applied={T|T < 7298994112305311744}]
I20260620 18:43:04.456787 29323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:04.456892 29323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:04.457854 29323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0ff4321a82148c4b09e4c91fefb4d9e: Sending DeleteTablet for 3 replicas of tablet f3899b3056a9459f93eb6abb06b4cffb
I20260620 18:43:04.458282 29448 tablet_service.cc:1514] Processing DeleteTablet for tablet f3899b3056a9459f93eb6abb06b4cffb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:04 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:53654
I20260620 18:43:04.458302 29716 tablet_service.cc:1514] Processing DeleteTablet for tablet f3899b3056a9459f93eb6abb06b4cffb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:04 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:58096
I20260620 18:43:04.458578 29822 tablet_replica.cc:330] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a: stopping tablet replica
I20260620 18:43:04.458709 29822 raft_consensus.cc:2229] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:04.458791 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 29385
I20260620 18:43:04.458770 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet f3899b3056a9459f93eb6abb06b4cffb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:04 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:35140
I20260620 18:43:04.458854 29822 raft_consensus.cc:2258] T f3899b3056a9459f93eb6abb06b4cffb P e7749da5db6f46a2a880a0a91caa976a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:04.459486 29824 tablet_replica.cc:330] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2: stopping tablet replica
I20260620 18:43:04.459661 29824 raft_consensus.cc:2229] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:43:04.459818 29824 raft_consensus.cc:2258] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:04.460191 29823 tablet_replica.cc:330] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938: stopping tablet replica
I20260620 18:43:04.460325 29823 raft_consensus.cc:2229] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:04.460450 29823 raft_consensus.cc:2258] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:04.460537 29824 ts_tablet_manager.cc:1905] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:04.460978 29823 ts_tablet_manager.cc:1905] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:04.462486 29824 ts_tablet_manager.cc:1918] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:04.462671 29824 log.cc:1196] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/f3899b3056a9459f93eb6abb06b4cffb
I20260620 18:43:04.462944 29823 ts_tablet_manager.cc:1918] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:04.463040 29823 log.cc:1196] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/f3899b3056a9459f93eb6abb06b4cffb
I20260620 18:43:04.463270 29824 ts_tablet_manager.cc:1939] T f3899b3056a9459f93eb6abb06b4cffb P 4da87a745a9b4740bb790e2f7e02a4b2: Deleting consensus metadata
I20260620 18:43:04.463488 29823 ts_tablet_manager.cc:1939] T f3899b3056a9459f93eb6abb06b4cffb P 487d44949ac84e0f97504d2965739938: Deleting consensus metadata
I20260620 18:43:04.463850 29307 catalog_manager.cc:4841] TS 4da87a745a9b4740bb790e2f7e02a4b2 (127.10.250.66:37501): tablet f3899b3056a9459f93eb6abb06b4cffb (table test-table [id=036af33cc4d54a838cc2d82b78f328ec]) successfully deleted
I20260620 18:43:04.463968 29309 catalog_manager.cc:4841] TS 487d44949ac84e0f97504d2965739938 (127.10.250.67:36009): tablet f3899b3056a9459f93eb6abb06b4cffb (table test-table [id=036af33cc4d54a838cc2d82b78f328ec]) successfully deleted
W20260620 18:43:04.466661 29309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:43001 (error 108)
I20260620 18:43:04.467341 29797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.467157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:43001 (local address 127.0.0.1:53666)
0620 18:43:04.467225 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:04.467300 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 18:43:04.467332 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 29519
W20260620 18:43:04.471272 29309 catalog_manager.cc:4584] TS e7749da5db6f46a2a880a0a91caa976a (127.10.250.65:43001): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3899b3056a9459f93eb6abb06b4cffb: Network error: Client connection negotiation failed: client connection to 127.10.250.65:43001: connect: Connection refused (error 111)
I20260620 18:43:04.473325 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 29653
I20260620 18:43:04.477905 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 29291
2026-06-20T18:43:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2142 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T18:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:04Z Disabled control of system clock
I20260620 18:43:04.506198 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:37357
--webserver_interface=127.10.250.126
--webserver_port=0
--builtin_ntp_servers=127.10.250.84:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:37357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:43:04.590500 29829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:04.590767 29829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:04.590790 29829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:04.592022 29829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:04.592070 29829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:04.592083 29829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:04.592095 29829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:04.592108 29829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:04.593497 29829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:37357
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:37357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:04.593725 29829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:04.593991 29829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:43:04.599480 29829 server_base.cc:1034] running on GCE node
W20260620 18:43:04.599440 29835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:04.599459 29837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:04.599473 29834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:04.599910 29829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:04.600273 29829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:04.601475 29829 hybrid_clock.cc:648] HybridClock initialized: now 1781980984601452 us; error 43 us; skew 500 ppm
I20260620 18:43:04.606005 29829 webserver.cc:460] Webserver started at http://127.10.250.126:35667/ using document root <none> and password file <none>
I20260620 18:43:04.606372 29829 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:04.606454 29829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:04.606647 29829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:04.607788 29829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "e4764dcce6314efdb5e1b157577c5251"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-wp6g"
server_key: "434def16965bdf45bfda640cdbe784ee"
server_key_iv: "71d1ff6a62216ee0f91ed25b487c673d"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.608191 29829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "e4764dcce6314efdb5e1b157577c5251"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-wp6g"
server_key: "434def16965bdf45bfda640cdbe784ee"
server_key_iv: "71d1ff6a62216ee0f91ed25b487c673d"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.609627 29829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:43:04.610610 29843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:04.610795 29829 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:04.610903 29829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "e4764dcce6314efdb5e1b157577c5251"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-wp6g"
server_key: "434def16965bdf45bfda640cdbe784ee"
server_key_iv: "71d1ff6a62216ee0f91ed25b487c673d"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.610989 29829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:04.634349 29829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:04.634768 29829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:04.634933 29829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:04.639420 29829 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:37357
I20260620 18:43:04.639467 29895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:37357 every 8 connection(s)
I20260620 18:43:04.639871 29829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:04.641029 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 29829
I20260620 18:43:04.641073 29896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:04.641161 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:04.641387 11241 external_mini_cluster.cc:1428] Setting key 6967c53cbc71f56f95f04e26f1cdaec4
I20260620 18:43:04.644912 29896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251: Bootstrap starting.
I20260620 18:43:04.645730 29896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:04.646067 29896 log.cc:824] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:04.646843 29896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251: No bootstrap required, opened a new log
I20260620 18:43:04.647711 29899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.641981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59086 (local address 127.10.250.126:37357)
0620 18:43:04.642245 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:04.642253 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:04.642281 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:43:04.642819 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:04.642826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:04.642895 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:04.643020 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:04.643359 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.644296 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.645466 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.645983 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.646082 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.646934 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:04.646939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:04.646939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:04.647152 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:04.647195 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:04.647209 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:04.647282 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:04.647374 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 18:43:04.647465 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":119,"threads_started":1}
I20260620 18:43:04.649242 29896 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4764dcce6314efdb5e1b157577c5251" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37357 } }
I20260620 18:43:04.649390 29896 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:04.649412 29896 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4764dcce6314efdb5e1b157577c5251, State: Initialized, Role: FOLLOWER
I20260620 18:43:04.649557 29896 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4764dcce6314efdb5e1b157577c5251" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37357 } }
I20260620 18:43:04.649641 29896 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:04.649673 29896 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:04.649727 29896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:04.650457 29896 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4764dcce6314efdb5e1b157577c5251" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37357 } }
I20260620 18:43:04.650602 29896 leader_election.cc:302] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4764dcce6314efdb5e1b157577c5251; no voters:
I20260620 18:43:04.650833 29896 leader_election.cc:288] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:04.650949 29901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:04.651139 29901 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [term 1 LEADER]: Becoming Leader. State: Replica: e4764dcce6314efdb5e1b157577c5251, State: Running, Role: LEADER
I20260620 18:43:04.651149 29896 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:04.651243 29901 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4764dcce6314efdb5e1b157577c5251" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37357 } }
I20260620 18:43:04.651719 29903 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4764dcce6314efdb5e1b157577c5251. Latest consensus state: current_term: 1 leader_uuid: "e4764dcce6314efdb5e1b157577c5251" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4764dcce6314efdb5e1b157577c5251" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37357 } } }
I20260620 18:43:04.651832 29903 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:04.651705 29902 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4764dcce6314efdb5e1b157577c5251" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4764dcce6314efdb5e1b157577c5251" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37357 } } }
I20260620 18:43:04.652002 29902 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251 [sys.catalog]: This master's current role is: LEADER
W20260620 18:43:04.652976 29917 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:43:04.653039 29917 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:43:04.653106 29910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:04.653307 29910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:04.654878 29910 catalog_manager.cc:1347] Generated new cluster ID: 88f56ba36952478c9b014659986d45fb
I20260620 18:43:04.654927 29910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:04.672976 29910 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:04.673573 29910 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:04.678912 29910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251: Generated new TSK 0
I20260620 18:43:04.679188 29910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:04.685658 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37357
--builtin_ntp_servers=127.10.250.84:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:43:04.776297 29920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:04.776625 29920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:04.776671 29920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:04.778146 29920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:04.778201 29920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:04.778230 29920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:43:04.779755 29920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:04.780021 29920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:04.780398 29920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:04.785712 29928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:04.785753 29926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:04.785956 29920 server_base.cc:1034] running on GCE node
W20260620 18:43:04.785779 29925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:04.786283 29920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:04.786608 29920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:04.787787 29920 hybrid_clock.cc:648] HybridClock initialized: now 1781980984787767 us; error 38 us; skew 500 ppm
I20260620 18:43:04.789247 29920 webserver.cc:460] Webserver started at http://127.10.250.65:39575/ using document root <none> and password file <none>
I20260620 18:43:04.789515 29920 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:04.789646 29920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:04.789805 29920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:04.791064 29920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "4bada0926a27490da62b24b1bfbb0018"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-wp6g"
server_key: "2f2765e773807fb2f25a8d20b5466258"
server_key_iv: "6d4af6a5cbb5d526a52c53eb137fca28"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.791460 29920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "4bada0926a27490da62b24b1bfbb0018"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-wp6g"
server_key: "2f2765e773807fb2f25a8d20b5466258"
server_key_iv: "6d4af6a5cbb5d526a52c53eb137fca28"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.792963 29920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:43:04.793889 29934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:04.794077 29920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:04.794214 29920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "4bada0926a27490da62b24b1bfbb0018"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-wp6g"
server_key: "2f2765e773807fb2f25a8d20b5466258"
server_key_iv: "6d4af6a5cbb5d526a52c53eb137fca28"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.794304 29920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:04.812311 29920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:04.812697 29920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:04.812866 29920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:04.813130 29920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:04.813484 29920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:04.813539 29920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:04.813585 29920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:04.813616 29920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:04.819744 29920 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:44729
I20260620 18:43:04.819792 30047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:44729 every 8 connection(s)
I20260620 18:43:04.820202 29920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:43:04.821085 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 29920
I20260620 18:43:04.821210 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:43:04.821462 11241 external_mini_cluster.cc:1428] Setting key 050d4fcd59aa5598d870a70a9f6c4872
I20260620 18:43:04.822659 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37357
--builtin_ntp_servers=127.10.250.84:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:43:04.828858 29899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.821213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:57581 (local address 127.10.250.126:37357)
0620 18:43:04.821654 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:04.821656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:04.821946 (+ 290us) server_negotiation.cc:407] Connection header received
0620 18:43:04.824154 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:04.824156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:04.824184 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:04.824251 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:04.825024 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.825854 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.827100 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.827492 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.827574 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.828490 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:04.828492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:04.828492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:04.828498 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:04.828516 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:04.828522 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:04.828587 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:04.828687 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 18:43:04.828723 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408}
I20260620 18:43:04.828976 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.821147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:37357 (local address 127.10.250.65:57581)
0620 18:43:04.821783 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:04.821833 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:04.824004 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:04.824336 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:04.824340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:04.824355 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:04.824900 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.824905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.825988 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.825990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.826996 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.826999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.827594 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.827595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.827723 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.828333 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:04.828334 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:04.828367 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:04.828394 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:04.828600 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:04.828603 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:04.828606 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:43:04.828663 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 18:43:04.828759 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":110,"threads_started":1}
I20260620 18:43:04.829573 30048 heartbeater.cc:347] Connected to a master server at 127.10.250.126:37357
I20260620 18:43:04.829744 30048 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:04.829962 30048 heartbeater.cc:511] Master 127.10.250.126:37357 requested a full tablet report, sending...
I20260620 18:43:04.830597 29860 ts_manager.cc:194] Registered new tserver with Master: 4bada0926a27490da62b24b1bfbb0018 (127.10.250.65:44729)
I20260620 18:43:04.831234 29860 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.250.65:57581
W20260620 18:43:04.909624 30051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:04.909863 30051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:04.909888 30051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:04.911244 30051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:04.911304 30051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:04.911334 30051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:43:04.913223 30051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:04.913583 30051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:04.913954 30051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:04.919684 30056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:04.919684 30057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:04.919829 30059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:04.920948 30051 server_base.cc:1034] running on GCE node
I20260620 18:43:04.921218 30051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:04.921535 30051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:04.922708 30051 hybrid_clock.cc:648] HybridClock initialized: now 1781980984922698 us; error 31 us; skew 500 ppm
I20260620 18:43:04.924122 30051 webserver.cc:460] Webserver started at http://127.10.250.66:41511/ using document root <none> and password file <none>
I20260620 18:43:04.924428 30051 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:04.924503 30051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:04.924669 30051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:04.925808 30051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "c11b97db235f4a5594c0d0601c1416c9"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-wp6g"
server_key: "41df4fdc3fae40515a9cf027701d99d7"
server_key_iv: "86ca7bdd56217e9664410d761ca73dee"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.926189 30051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "c11b97db235f4a5594c0d0601c1416c9"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-wp6g"
server_key: "41df4fdc3fae40515a9cf027701d99d7"
server_key_iv: "86ca7bdd56217e9664410d761ca73dee"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.927809 30051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:43:04.928687 30065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:04.928872 30051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:04.928967 30051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "c11b97db235f4a5594c0d0601c1416c9"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-wp6g"
server_key: "41df4fdc3fae40515a9cf027701d99d7"
server_key_iv: "86ca7bdd56217e9664410d761ca73dee"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.929051 30051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:04.943846 30051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:04.944209 30051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:04.944373 30051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:04.944629 30051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:04.944988 30051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:04.945043 30051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:04.945091 30051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:04.945115 30051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:04.951285 30051 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:45637
I20260620 18:43:04.951349 30178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:45637 every 8 connection(s)
I20260620 18:43:04.951683 30051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:43:04.957376 29899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.952552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:52209 (local address 127.10.250.126:37357)
0620 18:43:04.952633 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:04.952636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:04.952897 (+ 261us) server_negotiation.cc:407] Connection header received
0620 18:43:04.953477 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:04.953478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:04.953490 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:04.953551 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:04.954285 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.954878 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.955889 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.956110 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.956187 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.957080 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:04.957082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:04.957082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:04.957087 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:04.957104 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:04.957108 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:04.957160 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:04.957265 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 18:43:04.957293 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:43:04.957628 30181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.952517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:37357 (local address 127.10.250.66:52209)
0620 18:43:04.952711 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:04.952788 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:04.953356 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:04.953623 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:04.953628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:04.953638 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:04.954182 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.954187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.954985 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.954986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.955794 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:04.955797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.956197 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.956198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:04.956341 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.956931 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:04.956931 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:04.956961 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:04.956984 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:04.957170 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:04.957175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:04.957179 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:43:04.957268 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 18:43:04.957386 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260620 18:43:04.958145 30179 heartbeater.cc:347] Connected to a master server at 127.10.250.126:37357
I20260620 18:43:04.958226 30179 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:04.958436 30179 heartbeater.cc:511] Master 127.10.250.126:37357 requested a full tablet report, sending...
I20260620 18:43:04.958865 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 30051
I20260620 18:43:04.958874 29860 ts_manager.cc:194] Registered new tserver with Master: c11b97db235f4a5594c0d0601c1416c9 (127.10.250.66:45637)
I20260620 18:43:04.958972 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:43:04.959183 11241 external_mini_cluster.cc:1428] Setting key 6bf565f615846a7b70b6da0d5a37b3fd
I20260620 18:43:04.959342 29860 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.250.66:52209
I20260620 18:43:04.960202 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37357
--builtin_ntp_servers=127.10.250.84:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:43:05.043318 30182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:05.043572 30182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:05.043609 30182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:05.045691 30182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:05.045796 30182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:05.045847 30182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:43:05.047627 30182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:05.048061 30182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:05.048453 30182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:05.054325 30187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:05.054322 30188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:05.054561 30190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:05.054713 30182 server_base.cc:1034] running on GCE node
I20260620 18:43:05.054987 30182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:05.055284 30182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:05.056454 30182 hybrid_clock.cc:648] HybridClock initialized: now 1781980985056435 us; error 43 us; skew 500 ppm
I20260620 18:43:05.057934 30182 webserver.cc:460] Webserver started at http://127.10.250.67:33505/ using document root <none> and password file <none>
I20260620 18:43:05.058240 30182 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:05.058296 30182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:05.058444 30182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:05.059700 30182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "0f2591b6e7154ba8a6514a829346b575"
format_stamp: "Formatted at 2026-06-20 18:43:05 on dist-test-slave-wp6g"
server_key: "5697118a373fdb92560ca973cc298d3d"
server_key_iv: "a9ffc65b70008901d5c923885fd70019"
server_key_version: "encryptionkey@0"
I20260620 18:43:05.060130 30182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "0f2591b6e7154ba8a6514a829346b575"
format_stamp: "Formatted at 2026-06-20 18:43:05 on dist-test-slave-wp6g"
server_key: "5697118a373fdb92560ca973cc298d3d"
server_key_iv: "a9ffc65b70008901d5c923885fd70019"
server_key_version: "encryptionkey@0"
I20260620 18:43:05.061558 30182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:43:05.062424 30196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:05.062635 30182 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:05.062748 30182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "0f2591b6e7154ba8a6514a829346b575"
format_stamp: "Formatted at 2026-06-20 18:43:05 on dist-test-slave-wp6g"
server_key: "5697118a373fdb92560ca973cc298d3d"
server_key_iv: "a9ffc65b70008901d5c923885fd70019"
server_key_version: "encryptionkey@0"
I20260620 18:43:05.062847 30182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:05.081445 30182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:05.081802 30182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:05.081964 30182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:05.082262 30182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:05.082695 30182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:05.082749 30182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:05.082794 30182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:05.082821 30182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:05.089118 30182 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:34477
I20260620 18:43:05.089201 30309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:34477 every 8 connection(s)
I20260620 18:43:05.089552 30182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:43:05.095341 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 30182
I20260620 18:43:05.095381 29899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.090376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:43589 (local address 127.10.250.126:37357)
0620 18:43:05.090449 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:05.090451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:05.090771 (+ 320us) server_negotiation.cc:407] Connection header received
0620 18:43:05.091357 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:05.091359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:05.091372 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:05.091424 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:05.092192 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.092697 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.093735 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.093934 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.094022 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.095049 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:05.095051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:05.095051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:05.095058 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:05.095076 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:05.095080 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:05.095151 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:05.095258 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 18:43:05.095292 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:43:05.095566 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:43:05.095700 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.090382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:37357 (local address 127.10.250.67:43589)
0620 18:43:05.090620 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:05.090676 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:05.091228 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:05.091500 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:05.091504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:05.091514 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:05.092078 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.092084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.092799 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.092800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.093625 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.093628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.094034 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.094035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.094184 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.094831 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:05.094832 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:05.094865 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:05.094892 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:05.095165 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:05.095171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:05.095174 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:43:05.095262 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:43:05.095383 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20260620 18:43:05.095784 11241 external_mini_cluster.cc:1428] Setting key 7cbd3ba01d15f1b87c268359e603a717
I20260620 18:43:05.096498 30310 heartbeater.cc:347] Connected to a master server at 127.10.250.126:37357
I20260620 18:43:05.096825 30310 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:05.097116 30310 heartbeater.cc:511] Master 127.10.250.126:37357 requested a full tablet report, sending...
I20260620 18:43:05.097579 29859 ts_manager.cc:194] Registered new tserver with Master: 0f2591b6e7154ba8a6514a829346b575 (127.10.250.67:34477)
I20260620 18:43:05.097952 29859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.250.67:43589
I20260620 18:43:05.108294 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:43:05.113250 29899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.109938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59090 (local address 127.10.250.126:37357)
0620 18:43:05.109996 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:05.109999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:05.110014 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:43:05.110090 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:05.110092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:05.110104 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:05.110177 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:05.110366 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.111055 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.111859 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.112060 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.112140 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.112913 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:05.112915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:05.112917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:05.112936 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:05.112950 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:05.112955 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:05.113060 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:05.113136 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 18:43:05.113149 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 18:43:05.116037 29859 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:43:05.117700 29859 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:43:05.124563 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.120755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.10.250.66:45637)
0620 18:43:05.120929 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:05.120933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:05.120942 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:43:05.121001 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:05.121003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:05.121019 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:05.121125 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:05.121312 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.122195 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.123541 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.124120 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.124214 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.124333 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:05.124423 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 18:43:05.124472 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260620 18:43:05.124560 30319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.120376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:45637 (local address 127.0.0.1:46568)
0620 18:43:05.120560 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:05.120653 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:05.120696 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:05.121136 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:05.121139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:05.121141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:05.121226 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.121228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.122284 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.122286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.123445 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.123448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.124226 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.124227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.124315 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.124329 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 18:43:05.124407 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:43:05.124466 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":62,"threads_started":1}
I20260620 18:43:05.125113 30321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.120589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:34477 (local address 127.0.0.1:45492)
0620 18:43:05.121024 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:05.121035 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:05.121093 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:05.121478 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:05.121479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:05.121482 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:05.121527 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.121536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.122968 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.122969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.123862 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.123864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.124856 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.124857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.124922 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.124926 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:43:05.124985 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 18:43:05.125026 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":59,"threads_started":1}
I20260620 18:43:05.125142 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.121003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.10.250.67:34477)
0620 18:43:05.121303 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:05.121306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:05.121315 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:43:05.121356 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:05.121358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:05.121372 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:05.121474 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:05.121629 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.122868 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.123950 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.124765 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.124839 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.124921 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:05.125001 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 18:43:05.125046 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":86,"threads_started":1}
I20260620 18:43:05.125869 30113 tablet_service.cc:1467] Processing CreateTablet for tablet bf31687fd6db45bdb5d0cbc1018e6c78 (DEFAULT_TABLE table=test-table [id=c671d9e0375a4177b457575078669be6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:05.126186 30318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.120427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:44729 (local address 127.0.0.1:45096)
0620 18:43:05.120560 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:05.120606 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:05.120675 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:05.121452 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:05.121453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:05.121455 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:05.121504 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.121506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.123970 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.123971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.125005 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.125008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.125940 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.125941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.126003 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.126006 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:43:05.126048 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 18:43:05.126108 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260620 18:43:05.126191 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.120533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.10.250.65:44729)
0620 18:43:05.121214 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:05.121219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:05.121230 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:43:05.121276 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:05.121279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:05.121294 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:05.121374 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:05.121563 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.122865 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.125101 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.125851 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.125937 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.126021 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:05.126059 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 18:43:05.126112 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":119,"threads_started":1}
I20260620 18:43:05.126323 30113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf31687fd6db45bdb5d0cbc1018e6c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:05.126926 29982 tablet_service.cc:1467] Processing CreateTablet for tablet bf31687fd6db45bdb5d0cbc1018e6c78 (DEFAULT_TABLE table=test-table [id=c671d9e0375a4177b457575078669be6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:05.127266 29982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf31687fd6db45bdb5d0cbc1018e6c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:05.129367 30324 tablet_bootstrap.cc:492] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9: Bootstrap starting.
I20260620 18:43:05.129958 30324 tablet_bootstrap.cc:654] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:05.130299 30324 log.cc:824] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:05.130466 30325 tablet_bootstrap.cc:492] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018: Bootstrap starting.
I20260620 18:43:05.131158 30325 tablet_bootstrap.cc:654] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:05.131275 30324 tablet_bootstrap.cc:492] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9: No bootstrap required, opened a new log
I20260620 18:43:05.131361 30324 ts_tablet_manager.cc:1397] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:43:05.131311 30244 tablet_service.cc:1467] Processing CreateTablet for tablet bf31687fd6db45bdb5d0cbc1018e6c78 (DEFAULT_TABLE table=test-table [id=c671d9e0375a4177b457575078669be6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:05.131498 30325 log.cc:824] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:05.131709 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf31687fd6db45bdb5d0cbc1018e6c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:05.132509 30325 tablet_bootstrap.cc:492] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018: No bootstrap required, opened a new log
I20260620 18:43:05.132579 30325 ts_tablet_manager.cc:1397] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:43:05.134163 30324 raft_consensus.cc:348] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11b97db235f4a5594c0d0601c1416c9" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45637 } } peers { permanent_uuid: "4bada0926a27490da62b24b1bfbb0018" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 44729 } } peers { permanent_uuid: "0f2591b6e7154ba8a6514a829346b575" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 34477 } }
I20260620 18:43:05.134353 30324 raft_consensus.cc:374] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:05.134392 30324 raft_consensus.cc:729] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c11b97db235f4a5594c0d0601c1416c9, State: Initialized, Role: FOLLOWER
I20260620 18:43:05.134506 30324 consensus_queue.cc:260] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11b97db235f4a5594c0d0601c1416c9" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45637 } } peers { permanent_uuid: "4bada0926a27490da62b24b1bfbb0018" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 44729 } } peers { permanent_uuid: "0f2591b6e7154ba8a6514a829346b575" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 34477 } }
I20260620 18:43:05.134662 30325 raft_consensus.cc:348] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11b97db235f4a5594c0d0601c1416c9" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45637 } } peers { permanent_uuid: "4bada0926a27490da62b24b1bfbb0018" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 44729 } } peers { permanent_uuid: "0f2591b6e7154ba8a6514a829346b575" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 34477 } }
I20260620 18:43:05.134850 30324 ts_tablet_manager.cc:1428] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260620 18:43:05.134852 30325 raft_consensus.cc:374] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:05.134908 30325 raft_consensus.cc:729] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bada0926a27490da62b24b1bfbb0018, State: Initialized, Role: FOLLOWER
I20260620 18:43:05.135062 30179 heartbeater.cc:503] Master 127.10.250.126:37357 was elected leader, sending a full tablet report...
I20260620 18:43:05.135071 30325 consensus_queue.cc:260] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11b97db235f4a5594c0d0601c1416c9" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45637 } } peers { permanent_uuid: "4bada0926a27490da62b24b1bfbb0018" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 44729 } } peers { permanent_uuid: "0f2591b6e7154ba8a6514a829346b575" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 34477 } }
I20260620 18:43:05.135337 30325 ts_tablet_manager.cc:1428] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:43:05.135370 30330 tablet_bootstrap.cc:492] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575: Bootstrap starting.
I20260620 18:43:05.135519 30048 heartbeater.cc:503] Master 127.10.250.126:37357 was elected leader, sending a full tablet report...
I20260620 18:43:05.136241 30330 tablet_bootstrap.cc:654] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:05.136490 30330 log.cc:824] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:05.137267 30330 tablet_bootstrap.cc:492] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575: No bootstrap required, opened a new log
I20260620 18:43:05.137362 30330 ts_tablet_manager.cc:1397] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:43:05.139264 30330 raft_consensus.cc:348] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11b97db235f4a5594c0d0601c1416c9" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45637 } } peers { permanent_uuid: "4bada0926a27490da62b24b1bfbb0018" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 44729 } } peers { permanent_uuid: "0f2591b6e7154ba8a6514a829346b575" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 34477 } }
I20260620 18:43:05.139418 30330 raft_consensus.cc:374] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:05.139456 30330 raft_consensus.cc:729] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2591b6e7154ba8a6514a829346b575, State: Initialized, Role: FOLLOWER
I20260620 18:43:05.139593 30330 consensus_queue.cc:260] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11b97db235f4a5594c0d0601c1416c9" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45637 } } peers { permanent_uuid: "4bada0926a27490da62b24b1bfbb0018" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 44729 } } peers { permanent_uuid: "0f2591b6e7154ba8a6514a829346b575" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 34477 } }
I20260620 18:43:05.139858 30330 ts_tablet_manager.cc:1428] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 18:43:05.139948 30310 heartbeater.cc:503] Master 127.10.250.126:37357 was elected leader, sending a full tablet report...
I20260620 18:43:05.198644 30329 raft_consensus.cc:482] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:43:05.198887 30329 raft_consensus.cc:504] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11b97db235f4a5594c0d0601c1416c9" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45637 } } peers { permanent_uuid: "4bada0926a27490da62b24b1bfbb0018" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 44729 } } peers { permanent_uuid: "0f2591b6e7154ba8a6514a829346b575" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 34477 } }
I20260620 18:43:05.199294 30329 leader_election.cc:288] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c11b97db235f4a5594c0d0601c1416c9 (127.10.250.66:45637), 0f2591b6e7154ba8a6514a829346b575 (127.10.250.67:34477)
W20260620 18:43:05.202308 30180 tablet.cc:2369] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:05.203035 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.199644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:45637 (local address 127.10.250.65:49585)
0620 18:43:05.199797 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:05.199811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:05.199904 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:05.200304 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:05.200305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:05.200307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:05.200387 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.200389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.201081 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.201082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.201951 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.201953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.202748 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.202750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.202835 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.202838 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:43:05.202907 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 18:43:05.202964 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 18:43:05.203035 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.199727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:49585 (local address 127.10.250.66:45637)
0620 18:43:05.199807 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:05.199809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:05.200056 (+ 247us) server_negotiation.cc:407] Connection header received
0620 18:43:05.200096 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:05.200097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:05.200111 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:05.200242 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:05.200477 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.200980 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.202060 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.202638 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.202738 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.202748 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:05.202915 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 18:43:05.202962 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:43:05.203392 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.200216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:46737 (local address 127.10.250.67:34477)
0620 18:43:05.200273 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:05.200275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:05.200330 (+ 55us) server_negotiation.cc:407] Connection header received
0620 18:43:05.200413 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:05.200414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:05.200423 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:05.200498 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:05.200684 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.201287 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.202322 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.202839 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.202931 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.202939 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:05.203284 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 18:43:05.203330 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 18:43:05.203414 30133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf31687fd6db45bdb5d0cbc1018e6c78" candidate_uuid: "4bada0926a27490da62b24b1bfbb0018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c11b97db235f4a5594c0d0601c1416c9" is_pre_election: true
I20260620 18:43:05.203591 30133 raft_consensus.cc:2454] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bada0926a27490da62b24b1bfbb0018 in term 0.
I20260620 18:43:05.203657 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.199956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:34477 (local address 127.10.250.65:46737)
0620 18:43:05.200248 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:05.200260 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:05.200348 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:05.200511 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:05.200512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:05.200514 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:05.200588 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.200590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.201418 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.201419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.202225 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.202227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.202932 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.202933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.203031 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.203034 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:43:05.203518 (+ 484us) client_negotiation.cc:240] Negotiation successful
0620 18:43:05.203565 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":185,"threads_started":1}
I20260620 18:43:05.203890 29936 leader_election.cc:302] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bada0926a27490da62b24b1bfbb0018, c11b97db235f4a5594c0d0601c1416c9; no voters:
I20260620 18:43:05.204007 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf31687fd6db45bdb5d0cbc1018e6c78" candidate_uuid: "4bada0926a27490da62b24b1bfbb0018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2591b6e7154ba8a6514a829346b575" is_pre_election: true
I20260620 18:43:05.204092 30329 raft_consensus.cc:2781] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:43:05.204185 30329 raft_consensus.cc:482] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:43:05.204185 30264 raft_consensus.cc:2454] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bada0926a27490da62b24b1bfbb0018 in term 0.
I20260620 18:43:05.204221 30329 raft_consensus.cc:3037] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:05.205017 30329 raft_consensus.cc:504] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11b97db235f4a5594c0d0601c1416c9" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45637 } } peers { permanent_uuid: "4bada0926a27490da62b24b1bfbb0018" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 44729 } } peers { permanent_uuid: "0f2591b6e7154ba8a6514a829346b575" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 34477 } }
I20260620 18:43:05.205163 30329 leader_election.cc:288] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [CANDIDATE]: Term 1 election: Requested vote from peers c11b97db235f4a5594c0d0601c1416c9 (127.10.250.66:45637), 0f2591b6e7154ba8a6514a829346b575 (127.10.250.67:34477)
I20260620 18:43:05.205374 30133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf31687fd6db45bdb5d0cbc1018e6c78" candidate_uuid: "4bada0926a27490da62b24b1bfbb0018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c11b97db235f4a5594c0d0601c1416c9"
I20260620 18:43:05.205403 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf31687fd6db45bdb5d0cbc1018e6c78" candidate_uuid: "4bada0926a27490da62b24b1bfbb0018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2591b6e7154ba8a6514a829346b575"
I20260620 18:43:05.205493 30264 raft_consensus.cc:3037] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:05.205493 30133 raft_consensus.cc:3037] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:05.206594 30264 raft_consensus.cc:2454] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bada0926a27490da62b24b1bfbb0018 in term 1.
I20260620 18:43:05.206588 30133 raft_consensus.cc:2454] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bada0926a27490da62b24b1bfbb0018 in term 1.
I20260620 18:43:05.206792 29936 leader_election.cc:302] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f2591b6e7154ba8a6514a829346b575, 4bada0926a27490da62b24b1bfbb0018; no voters:
I20260620 18:43:05.206907 30329 raft_consensus.cc:2781] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:05.207095 30329 raft_consensus.cc:686] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [term 1 LEADER]: Becoming Leader. State: Replica: 4bada0926a27490da62b24b1bfbb0018, State: Running, Role: LEADER
I20260620 18:43:05.207227 30329 consensus_queue.cc:237] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11b97db235f4a5594c0d0601c1416c9" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45637 } } peers { permanent_uuid: "4bada0926a27490da62b24b1bfbb0018" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 44729 } } peers { permanent_uuid: "0f2591b6e7154ba8a6514a829346b575" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 34477 } }
I20260620 18:43:05.208407 29860 catalog_manager.cc:5495] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bada0926a27490da62b24b1bfbb0018 (127.10.250.65). New cstate: current_term: 1 leader_uuid: "4bada0926a27490da62b24b1bfbb0018" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11b97db235f4a5594c0d0601c1416c9" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bada0926a27490da62b24b1bfbb0018" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 44729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f2591b6e7154ba8a6514a829346b575" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260620 18:43:05.220219 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.217383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45106 (local address 127.10.250.65:44729)
0620 18:43:05.217456 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:05.217459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:05.217471 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:43:05.217527 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:05.217528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:05.217550 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:05.217716 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:05.217853 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.218377 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.219297 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.219517 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.219624 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.219789 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:05.219918 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:05.220041 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:05.220076 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 18:43:05.220089 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:43:05.222448 30264 raft_consensus.cc:1261] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575 [term 1 FOLLOWER]: Refusing update from remote peer 4bada0926a27490da62b24b1bfbb0018: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:05.222462 30133 raft_consensus.cc:1261] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9 [term 1 FOLLOWER]: Refusing update from remote peer 4bada0926a27490da62b24b1bfbb0018: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:05.222991 30329 consensus_queue.cc:1035] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c11b97db235f4a5594c0d0601c1416c9" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:05.223204 30334 consensus_queue.cc:1035] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2591b6e7154ba8a6514a829346b575" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:05.225780 30337 mvcc.cc:205] Tried to move back new op lower bound from 7298994115467468800 to 7298994115409690624. Current Snapshot: MvccSnapshot[applied={T|T < 7298994115467468800}]
I20260620 18:43:05.226562 30339 mvcc.cc:205] Tried to move back new op lower bound from 7298994115467468800 to 7298994115409690624. Current Snapshot: MvccSnapshot[applied={T|T < 7298994115467468800}]
I20260620 18:43:05.226701 30338 mvcc.cc:205] Tried to move back new op lower bound from 7298994115467468800 to 7298994115409690624. Current Snapshot: MvccSnapshot[applied={T|T < 7298994115467468800}]
I20260620 18:43:05.228479 29859 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:05.228578 29859 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:05.229408 29859 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4764dcce6314efdb5e1b157577c5251: Sending DeleteTablet for 3 replicas of tablet bf31687fd6db45bdb5d0cbc1018e6c78
I20260620 18:43:05.229744 30113 tablet_service.cc:1514] Processing DeleteTablet for tablet bf31687fd6db45bdb5d0cbc1018e6c78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:05 UTC) from {username='slave'} at 127.0.0.1:46568
I20260620 18:43:05.229758 29982 tablet_service.cc:1514] Processing DeleteTablet for tablet bf31687fd6db45bdb5d0cbc1018e6c78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:05 UTC) from {username='slave'} at 127.0.0.1:45096
I20260620 18:43:05.229784 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet bf31687fd6db45bdb5d0cbc1018e6c78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:05 UTC) from {username='slave'} at 127.0.0.1:45492
I20260620 18:43:05.229997 30347 tablet_replica.cc:330] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575: stopping tablet replica
I20260620 18:43:05.230106 30347 raft_consensus.cc:2229] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:05.230233 30347 raft_consensus.cc:2258] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:05.230261 30348 tablet_replica.cc:330] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018: stopping tablet replica
I20260620 18:43:05.230438 30348 raft_consensus.cc:2229] T bf31687fd6db45bdb5d0cbc1018e6c78 P 4bada0926a27490da62b24b1bfbb0018 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:43:05.230450 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 29920
I20260620 18:43:05.230722 30347 ts_tablet_manager.cc:1905] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:05.232942 30347 ts_tablet_manager.cc:1918] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:05.233047 30347 log.cc:1196] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/bf31687fd6db45bdb5d0cbc1018e6c78
I20260620 18:43:05.233316 30347 ts_tablet_manager.cc:1939] T bf31687fd6db45bdb5d0cbc1018e6c78 P 0f2591b6e7154ba8a6514a829346b575: Deleting consensus metadata
I20260620 18:43:05.233793 29845 catalog_manager.cc:4841] TS 0f2591b6e7154ba8a6514a829346b575 (127.10.250.67:34477): tablet bf31687fd6db45bdb5d0cbc1018e6c78 (table test-table [id=c671d9e0375a4177b457575078669be6]) successfully deleted
I20260620 18:43:05.234723 30346 tablet_replica.cc:330] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9: stopping tablet replica
I20260620 18:43:05.234886 30346 raft_consensus.cc:2229] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:05.235037 30346 raft_consensus.cc:2258] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:05.235509 30346 ts_tablet_manager.cc:1905] T bf31687fd6db45bdb5d0cbc1018e6c78 P c11b97db235f4a5594c0d0601c1416c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:05.236704 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 30051
W20260620 18:43:05.236776 29846 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:44729 (error 108)
I20260620 18:43:05.242296 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 30182
I20260620 18:43:05.242560 30321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.242346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:45637 (local address 127.0.0.1:46572)
0620 18:43:05.242412 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:05.242466 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.66:45637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":20}
I20260620 18:43:05.243089 30318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.242328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:44729 (local address 127.0.0.1:45110)
0620 18:43:05.242410 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:05.242470 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 18:43:05.243299 29846 catalog_manager.cc:4584] TS 4bada0926a27490da62b24b1bfbb0018 (127.10.250.65:44729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf31687fd6db45bdb5d0cbc1018e6c78: Network error: Client connection negotiation failed: client connection to 127.10.250.65:44729: connect: Connection refused (error 111)
I20260620 18:43:05.248178 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 29829
2026-06-20T18:43:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (769 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:05 dist-test-slave-wp6g krb5kdc[30352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:05 dist-test-slave-wp6g krb5kdc[30352](info): set up 2 sockets
Jun 20 18:43:05 dist-test-slave-wp6g krb5kdc[30352](info): commencing operation
krb5kdc: starting...
W20260620 18:43:06.322410 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:06 dist-test-slave-wp6g krb5kdc[30352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:43:06.453076 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38173
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:45587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:38173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:06.539886 30365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:06.540187 30365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:06.540234 30365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:06.541508 30365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:06.541587 30365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:06.541612 30365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:06.541635 30365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:06.541662 30365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:06.543320 30365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:45587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:38173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:38173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:06.543680 30365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:06.543942 30365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:06.550232 30371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:06.550437 30370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:06.550514 30373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:06.550675 30365 server_base.cc:1034] running on GCE node
I20260620 18:43:06.550976 30365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:06.551375 30365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:06.552559 30365 hybrid_clock.cc:648] HybridClock initialized: now 1781980986552541 us; error 30 us; skew 500 ppm
Jun 20 18:43:06 dist-test-slave-wp6g krb5kdc[30352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980986, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:06.555553 30365 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:43:06.556092 30365 webserver.cc:460] Webserver started at http://127.10.250.126:45613/ using document root <none> and password file <none>
I20260620 18:43:06.556354 30365 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:06.556406 30365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:06.556545 30365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:06.557732 30365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "05cb6b9a858c4bb09ce393f23aa4bde1"
format_stamp: "Formatted at 2026-06-20 18:43:06 on dist-test-slave-wp6g"
server_key: "b5089be7a7df1a42ec71a477b09e573e"
server_key_iv: "43ba9ff40ccaee870537c5742c4aef04"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.558118 30365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "05cb6b9a858c4bb09ce393f23aa4bde1"
format_stamp: "Formatted at 2026-06-20 18:43:06 on dist-test-slave-wp6g"
server_key: "b5089be7a7df1a42ec71a477b09e573e"
server_key_iv: "43ba9ff40ccaee870537c5742c4aef04"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.559891 30365 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 18:43:06.560895 30380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:06.561208 30365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:06.561307 30365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "05cb6b9a858c4bb09ce393f23aa4bde1"
format_stamp: "Formatted at 2026-06-20 18:43:06 on dist-test-slave-wp6g"
server_key: "b5089be7a7df1a42ec71a477b09e573e"
server_key_iv: "43ba9ff40ccaee870537c5742c4aef04"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.561430 30365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:06.573175 30365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:06.576406 30365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:06.576848 30365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:06.581684 30365 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:38173
I20260620 18:43:06.581715 30432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:38173 every 8 connection(s)
I20260620 18:43:06.582139 30365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:06.583257 30433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:06.585734 30433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1: Bootstrap starting.
I20260620 18:43:06.586350 30433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:06.586704 30433 log.cc:824] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:06.587502 30433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1: No bootstrap required, opened a new log
I20260620 18:43:06.589035 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 30365
I20260620 18:43:06.589161 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:06.589341 11241 external_mini_cluster.cc:1428] Setting key 9f22b1cd8df53068c65b8e5d9ab47d14
I20260620 18:43:06.589589 30433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cb6b9a858c4bb09ce393f23aa4bde1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38173 } }
I20260620 18:43:06.589798 30433 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:06.589844 30433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05cb6b9a858c4bb09ce393f23aa4bde1, State: Initialized, Role: FOLLOWER
I20260620 18:43:06.589979 30433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cb6b9a858c4bb09ce393f23aa4bde1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38173 } }
I20260620 18:43:06.590068 30433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:06.590111 30433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:06.590170 30433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:06.591454 30433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cb6b9a858c4bb09ce393f23aa4bde1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38173 } }
I20260620 18:43:06.591580 30433 leader_election.cc:302] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05cb6b9a858c4bb09ce393f23aa4bde1; no voters:
I20260620 18:43:06.591768 30433 leader_election.cc:288] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:06.591890 30438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:06.592042 30433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:06.592088 30438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [term 1 LEADER]: Becoming Leader. State: Replica: 05cb6b9a858c4bb09ce393f23aa4bde1, State: Running, Role: LEADER
I20260620 18:43:06.592206 30438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cb6b9a858c4bb09ce393f23aa4bde1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38173 } }
I20260620 18:43:06.592614 30438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05cb6b9a858c4bb09ce393f23aa4bde1. Latest consensus state: current_term: 1 leader_uuid: "05cb6b9a858c4bb09ce393f23aa4bde1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cb6b9a858c4bb09ce393f23aa4bde1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38173 } } }
I20260620 18:43:06.592624 30440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05cb6b9a858c4bb09ce393f23aa4bde1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cb6b9a858c4bb09ce393f23aa4bde1" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 38173 } } }
I20260620 18:43:06.592733 30440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:06.592895 30438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:06.593178 30447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:06.594048 30447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:06.596082 30447 catalog_manager.cc:1347] Generated new cluster ID: b23b6666ce6a42fda42d30b97a7bfda0
I20260620 18:43:06.596159 30447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:43:06 dist-test-slave-wp6g krb5kdc[30352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:06.602571 30447 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:06.602485 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.589933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.10.250.126:38173)
0620 18:43:06.590169 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:06.590180 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:06.590217 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:43:06.590372 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:06.590381 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:06.590439 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:06.590632 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:06.591258 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.592698 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.594041 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.594672 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.594802 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.598193 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:06.598199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:06.598199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:06.598214 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:06.600457 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:06.600925 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:06.600929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:06.600930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:06.601080 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:06.601335 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:06.601336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:06.601336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:06.601716 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:06.601893 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:06.602089 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 18:43:06.602254 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
I20260620 18:43:06.603199 30447 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:06.606249 30447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1: Generated new TSK 0
I20260620 18:43:06.606493 30447 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.250.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
Entry for principal kudu/127.10.250.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab.
I20260620 18:43:06.639528 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38173
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:45587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:06.730322 30459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:06.730690 30459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:06.730741 30459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:06.732136 30459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:06.732225 30459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:06.732277 30459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:43:06.734071 30459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:45587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.65.keytab
--principal=kudu/127.10.250.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:06.734484 30459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:06.734864 30459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:06.741333 30467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:06.741340 30464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:06.741602 30459 server_base.cc:1034] running on GCE node
W20260620 18:43:06.741356 30465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:06.742014 30459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:06.742312 30459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:06.743489 30459 hybrid_clock.cc:648] HybridClock initialized: now 1781980986743465 us; error 42 us; skew 500 ppm
Jun 20 18:43:06 dist-test-slave-wp6g krb5kdc[30352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980986, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:06.746613 30459 init.cc:377] Logged in from keytab as kudu/127.10.250.65@KRBTEST.COM (short username kudu)
I20260620 18:43:06.747143 30459 webserver.cc:460] Webserver started at http://127.10.250.65:36577/ using document root <none> and password file <none>
I20260620 18:43:06.747388 30459 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:06.747457 30459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:06.747618 30459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:06.748880 30459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "14a70e196655445489a2f66d1d052f5f"
format_stamp: "Formatted at 2026-06-20 18:43:06 on dist-test-slave-wp6g"
server_key: "08f907b6eca36a32d73caafc63cf9063"
server_key_iv: "6f0a5f333e7bd61b6f3422be039d1e7e"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.749320 30459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "14a70e196655445489a2f66d1d052f5f"
format_stamp: "Formatted at 2026-06-20 18:43:06 on dist-test-slave-wp6g"
server_key: "08f907b6eca36a32d73caafc63cf9063"
server_key_iv: "6f0a5f333e7bd61b6f3422be039d1e7e"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.751062 30459 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 18:43:06.752110 30474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:06.752408 30459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:06.752588 30459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "14a70e196655445489a2f66d1d052f5f"
format_stamp: "Formatted at 2026-06-20 18:43:06 on dist-test-slave-wp6g"
server_key: "08f907b6eca36a32d73caafc63cf9063"
server_key_iv: "6f0a5f333e7bd61b6f3422be039d1e7e"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.752686 30459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:06.764168 30459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:06.767222 30459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:06.767441 30459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:06.767738 30459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:06.768146 30459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:06.768206 30459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:06.768247 30459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:06.768296 30459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:06.775529 30459 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:37349
I20260620 18:43:06.775633 30587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:37349 every 8 connection(s)
I20260620 18:43:06.776016 30459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:43:06 dist-test-slave-wp6g krb5kdc[30352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980986, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.65@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:06.786146 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 30459
I20260620 18:43:06.786273 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:43:06.786484 11241 external_mini_cluster.cc:1428] Setting key 22d32d9cc6894018fd1680d649e5ba49
I20260620 18:43:06.786945 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.777088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:41005 (local address 127.10.250.126:38173)
0620 18:43:06.777190 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:06.777193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:06.777428 (+ 235us) server_negotiation.cc:407] Connection header received
0620 18:43:06.778069 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:06.778071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:06.778086 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:06.778154 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:06.779289 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.779832 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.780926 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.781156 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.781249 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.783493 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:06.783494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:06.783495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:06.783502 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:06.784973 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:06.785439 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:06.785440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:06.785441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:06.785559 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:06.785894 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:06.785915 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:06.785915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:06.786054 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:06.786116 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:06.786697 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 18:43:06.786823 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:43:06.787076 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.777021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38173 (local address 127.10.250.65:41005)
0620 18:43:06.777268 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:06.777313 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:06.777930 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:06.778234 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:06.778238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:06.778953 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:06.779158 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.779164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.779946 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.779947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.780803 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.780805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.781254 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.781255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.781362 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.781956 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:06.781965 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:06.783389 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:06.785079 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:06.785086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:06.785088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:06.785346 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:06.785684 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:06.785685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:06.785686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:06.785819 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:06.786149 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:06.786158 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:06.786389 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 18:43:06.786579 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 18:43:06.786806 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
I20260620 18:43:06.787621 30588 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38173
I20260620 18:43:06.787722 30588 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:06.787935 30588 heartbeater.cc:511] Master 127.10.250.126:38173 requested a full tablet report, sending...
I20260620 18:43:06.788548 30397 ts_manager.cc:194] Registered new tserver with Master: 14a70e196655445489a2f66d1d052f5f (127.10.250.65:37349)
I20260620 18:43:06.789314 30397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.65@KRBTEST.COM'} at 127.10.250.65:41005
WARNING: no policy specified for kudu/127.10.250.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
Entry for principal kudu/127.10.250.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab.
I20260620 18:43:06.814379 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38173
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:45587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:06.905467 30593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:06.905710 30593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:06.905738 30593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:06.907083 30593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:06.907136 30593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:06.907166 30593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:43:06.908689 30593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:45587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.66.keytab
--principal=kudu/127.10.250.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:06.908931 30593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:06.909132 30593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:06.914872 30601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:06.914932 30598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:06.915001 30599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:06.915088 30593 server_base.cc:1034] running on GCE node
I20260620 18:43:06.915367 30593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:06.915602 30593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:06.916791 30593 hybrid_clock.cc:648] HybridClock initialized: now 1781980986916776 us; error 30 us; skew 500 ppm
Jun 20 18:43:06 dist-test-slave-wp6g krb5kdc[30352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980986, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:06.919695 30593 init.cc:377] Logged in from keytab as kudu/127.10.250.66@KRBTEST.COM (short username kudu)
I20260620 18:43:06.920269 30593 webserver.cc:460] Webserver started at http://127.10.250.66:42487/ using document root <none> and password file <none>
I20260620 18:43:06.920542 30593 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:06.920589 30593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:06.920728 30593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:06.921856 30593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "06715a3399dd45d38a17b1822093c930"
format_stamp: "Formatted at 2026-06-20 18:43:06 on dist-test-slave-wp6g"
server_key: "7e44ffaff24008c0e052b8e668ff195b"
server_key_iv: "c902bfaec61482e977cfcf6db5de9f4d"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.922230 30593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "06715a3399dd45d38a17b1822093c930"
format_stamp: "Formatted at 2026-06-20 18:43:06 on dist-test-slave-wp6g"
server_key: "7e44ffaff24008c0e052b8e668ff195b"
server_key_iv: "c902bfaec61482e977cfcf6db5de9f4d"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.923745 30593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:43:06.924751 30608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:06.925103 30593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:06.925190 30593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "06715a3399dd45d38a17b1822093c930"
format_stamp: "Formatted at 2026-06-20 18:43:06 on dist-test-slave-wp6g"
server_key: "7e44ffaff24008c0e052b8e668ff195b"
server_key_iv: "c902bfaec61482e977cfcf6db5de9f4d"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.925305 30593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:06.932538 30593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:06.935134 30593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:06.935302 30593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:06.935570 30593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:06.935897 30593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:06.935930 30593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:06.935974 30593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:06.936002 30593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:06.942304 30593 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:45883
I20260620 18:43:06.942351 30721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:45883 every 8 connection(s)
I20260620 18:43:06.942759 30593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:43:06 dist-test-slave-wp6g krb5kdc[30352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980986, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.66@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:06.950060 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 30593
I20260620 18:43:06.950202 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:43:06.950403 11241 external_mini_cluster.cc:1428] Setting key 546ed585d86a22eaca7892cc42d53371
I20260620 18:43:06.953245 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.943592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:48853 (local address 127.10.250.126:38173)
0620 18:43:06.943638 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:06.943640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:06.943844 (+ 204us) server_negotiation.cc:407] Connection header received
0620 18:43:06.944348 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:06.944349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:06.944357 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:06.944416 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:06.945527 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.946024 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.947097 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.947284 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.947339 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.949598 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:06.949599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:06.949600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:06.949607 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:06.951429 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:06.951986 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:06.951987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:06.952012 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:06.952072 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:06.952408 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:06.952410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:06.952410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:06.952544 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:06.952620 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:06.953017 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 18:43:06.953121 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:43:06.953290 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.943558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38173 (local address 127.10.250.66:48853)
0620 18:43:06.943719 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:06.943763 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:06.944261 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:06.944472 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:06.944475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:06.945215 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:06.945422 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.945428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.946118 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.946119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.947018 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.947021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.947362 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.947363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.947477 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.948053 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:06.948061 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:06.949482 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:06.951542 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:06.951556 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:06.951559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:06.951906 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:06.952143 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:06.952143 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:06.952144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:06.952307 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:06.952628 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:06.952635 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:06.952823 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 18:43:06.952959 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 18:43:06.953117 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260620 18:43:06.954147 30722 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38173
I20260620 18:43:06.954264 30722 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:06.954466 30722 heartbeater.cc:511] Master 127.10.250.126:38173 requested a full tablet report, sending...
I20260620 18:43:06.954983 30397 ts_manager.cc:194] Registered new tserver with Master: 06715a3399dd45d38a17b1822093c930 (127.10.250.66:45883)
I20260620 18:43:06.955545 30397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.66@KRBTEST.COM'} at 127.10.250.66:48853
WARNING: no policy specified for kudu/127.10.250.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
Entry for principal kudu/127.10.250.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab.
I20260620 18:43:06.980049 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38173
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:45587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:07.069662 30727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:07.069909 30727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:07.069936 30727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:07.071597 30727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:07.071671 30727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:07.071708 30727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:43:07.073531 30727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:45587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.67.keytab
--principal=kudu/127.10.250.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:38173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:07.073793 30727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:07.074010 30727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:07.079603 30733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:07.079617 30735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:07.079617 30732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:07.080061 30727 server_base.cc:1034] running on GCE node
I20260620 18:43:07.080338 30727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:07.080618 30727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:07.081766 30727 hybrid_clock.cc:648] HybridClock initialized: now 1781980987081739 us; error 42 us; skew 500 ppm
Jun 20 18:43:07 dist-test-slave-wp6g krb5kdc[30352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980987, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:07.084872 30727 init.cc:377] Logged in from keytab as kudu/127.10.250.67@KRBTEST.COM (short username kudu)
I20260620 18:43:07.085366 30727 webserver.cc:460] Webserver started at http://127.10.250.67:41633/ using document root <none> and password file <none>
I20260620 18:43:07.085635 30727 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:07.085707 30727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:07.085865 30727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:07.087040 30727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "781bfab4f65e45d6b7294d72e5e8db92"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-wp6g"
server_key: "a1ef6cc3e76add594bb25286d7e5fd61"
server_key_iv: "704277777a78b0dca8ef8ecf1c073622"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.087414 30727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "781bfab4f65e45d6b7294d72e5e8db92"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-wp6g"
server_key: "a1ef6cc3e76add594bb25286d7e5fd61"
server_key_iv: "704277777a78b0dca8ef8ecf1c073622"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.088899 30727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:43:07.089778 30742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.089972 30727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:07.090135 30727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "781bfab4f65e45d6b7294d72e5e8db92"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-wp6g"
server_key: "a1ef6cc3e76add594bb25286d7e5fd61"
server_key_iv: "704277777a78b0dca8ef8ecf1c073622"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.090253 30727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:07.130932 30727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:07.133530 30727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:07.133718 30727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:07.133976 30727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:07.134325 30727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:07.134379 30727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.134421 30727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:07.134450 30727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.140836 30727 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:37951
I20260620 18:43:07.140908 30855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:37951 every 8 connection(s)
I20260620 18:43:07.141299 30727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:43:07.146138 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 30727
I20260620 18:43:07.146268 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:43:07.146464 11241 external_mini_cluster.cc:1428] Setting key 8bc546e9cd40f773619878acfdcfd74b
Jun 20 18:43:07 dist-test-slave-wp6g krb5kdc[30352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980987, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.67@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:07.153198 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.142179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:41291 (local address 127.10.250.126:38173)
0620 18:43:07.142263 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.142266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.142481 (+ 215us) server_negotiation.cc:407] Connection header received
0620 18:43:07.143080 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.143081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.143093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.143158 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:07.144418 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.144976 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.146320 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.146673 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.146758 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.149693 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:07.149694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:07.149695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:07.149703 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:07.151453 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:07.152169 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:07.152170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:07.152171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:07.152237 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:07.152499 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:07.152500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:07.152500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:07.152624 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:07.152704 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.153028 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.153125 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:43:07.153343 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.142115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:38173 (local address 127.10.250.67:41291)
0620 18:43:07.142308 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.142355 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:07.142958 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:07.143230 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:07.143234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:07.144049 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:07.144285 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.144291 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.145092 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.145093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.146215 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.146219 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.146832 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.146833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.147023 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.147758 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:07.147770 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:07.149534 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:07.151609 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:07.151620 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:07.151622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:07.152083 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:07.152307 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:07.152308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:07.152309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:07.152429 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:07.152706 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:07.152711 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:07.152861 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 18:43:07.152952 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 18:43:07.153099 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
I20260620 18:43:07.153970 30856 heartbeater.cc:347] Connected to a master server at 127.10.250.126:38173
I20260620 18:43:07.154090 30856 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:07.154343 30856 heartbeater.cc:511] Master 127.10.250.126:38173 requested a full tablet report, sending...
I20260620 18:43:07.154825 30397 ts_manager.cc:194] Registered new tserver with Master: 781bfab4f65e45d6b7294d72e5e8db92 (127.10.250.67:37951)
I20260620 18:43:07.155411 30397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.250.67@KRBTEST.COM'} at 127.10.250.67:41291
I20260620 18:43:07.158385 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:43:07.166783 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.159921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.10.250.126:38173)
0620 18:43:07.159981 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.159983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.160063 (+ 80us) server_negotiation.cc:407] Connection header received
0620 18:43:07.160114 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.160115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.160124 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.160177 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:07.160730 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.161247 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.162045 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.162252 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.162323 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.163822 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:07.163824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:07.163824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:07.163831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:07.165416 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:07.165924 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:07.165925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:07.165926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:07.165994 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:07.166200 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:07.166201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:07.166201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:07.166363 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:07.166443 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.166636 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.166710 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 18:43:07.168874 30397 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:43:07.170640 30397 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:43:07.177871 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.173751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60110 (local address 127.10.250.67:37951)
0620 18:43:07.173959 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.173964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.173972 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:43:07.174005 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.174007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.174024 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.174096 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:07.174301 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.175520 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.176830 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.177383 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.177480 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.177548 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.177731 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.177775 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":93,"threads_started":1}
I20260620 18:43:07.177874 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.173422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:45883 (local address 127.0.0.1:43716)
0620 18:43:07.174049 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.174059 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:07.174103 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:07.174404 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:07.174405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:07.174407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:07.174442 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.174443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.175707 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.175709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.176749 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.176752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.177640 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.177641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.177696 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.177699 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:43:07.177727 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 18:43:07.177763 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":35,"thread_start_us":107,"threads_started":1}
I20260620 18:43:07.178050 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.173693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.10.250.65:37349)
0620 18:43:07.173911 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.173918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.173929 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:43:07.173975 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.173978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.173996 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.174288 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:07.174392 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.175172 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.176658 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.177462 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.177812 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.177890 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.177917 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.177963 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
I20260620 18:43:07.178249 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.173882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43716 (local address 127.10.250.66:45883)
0620 18:43:07.174235 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.174240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.174251 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:43:07.174293 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.174296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.174310 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.174394 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:07.174504 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.175588 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.176852 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.177545 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.178012 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.178081 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.178111 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.178162 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":94,"threads_started":1}
I20260620 18:43:07.177884 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.173422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:37349 (local address 127.0.0.1:56322)
0620 18:43:07.173694 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.173706 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:07.173761 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:07.174108 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:07.174112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:07.174114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:07.174207 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.174208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.175285 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.175286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.176553 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.176556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.177573 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.177574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.177648 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.177653 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:43:07.177683 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 18:43:07.177757 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":56,"threads_started":1}
I20260620 18:43:07.178656 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.173420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:37951 (local address 127.0.0.1:60110)
0620 18:43:07.173614 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.173671 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:07.173748 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:07.174165 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:07.174165 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:07.174167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:07.174207 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.174208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.175642 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.175643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.176746 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.176748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.177483 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.177484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.177579 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.177595 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 18:43:07.177711 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:43:07.177764 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":41,"thread_start_us":117,"threads_started":1}
I20260620 18:43:07.178989 30656 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc2559edd5c4b67b87638ba22874850 (DEFAULT_TABLE table=test-table [id=9f6c10fa12e24787817642099ab6c6de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:07.179258 30522 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc2559edd5c4b67b87638ba22874850 (DEFAULT_TABLE table=test-table [id=9f6c10fa12e24787817642099ab6c6de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:07.179390 30656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc2559edd5c4b67b87638ba22874850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:07.179368 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc2559edd5c4b67b87638ba22874850 (DEFAULT_TABLE table=test-table [id=9f6c10fa12e24787817642099ab6c6de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:07.179565 30522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc2559edd5c4b67b87638ba22874850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:07.179714 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc2559edd5c4b67b87638ba22874850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:07.182438 30873 tablet_bootstrap.cc:492] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92: Bootstrap starting.
I20260620 18:43:07.183115 30874 tablet_bootstrap.cc:492] T 0fc2559edd5c4b67b87638ba22874850 P 14a70e196655445489a2f66d1d052f5f: Bootstrap starting.
I20260620 18:43:07.183334 30873 tablet_bootstrap.cc:654] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:07.183876 30874 tablet_bootstrap.cc:654] T 0fc2559edd5c4b67b87638ba22874850 P 14a70e196655445489a2f66d1d052f5f: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:07.183962 30873 log.cc:824] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:07.184463 30874 log.cc:824] T 0fc2559edd5c4b67b87638ba22874850 P 14a70e196655445489a2f66d1d052f5f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:07.184888 30875 tablet_bootstrap.cc:492] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930: Bootstrap starting.
I20260620 18:43:07.185283 30873 tablet_bootstrap.cc:492] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92: No bootstrap required, opened a new log
I20260620 18:43:07.185367 30873 ts_tablet_manager.cc:1397] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:43:07.185508 30874 tablet_bootstrap.cc:492] T 0fc2559edd5c4b67b87638ba22874850 P 14a70e196655445489a2f66d1d052f5f: No bootstrap required, opened a new log
I20260620 18:43:07.185601 30874 ts_tablet_manager.cc:1397] T 0fc2559edd5c4b67b87638ba22874850 P 14a70e196655445489a2f66d1d052f5f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:43:07.185611 30875 tablet_bootstrap.cc:654] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:07.186218 30875 log.cc:824] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:07.187161 30875 tablet_bootstrap.cc:492] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930: No bootstrap required, opened a new log
I20260620 18:43:07.187251 30875 ts_tablet_manager.cc:1397] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:43:07.187513 30874 raft_consensus.cc:348] T 0fc2559edd5c4b67b87638ba22874850 P 14a70e196655445489a2f66d1d052f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06715a3399dd45d38a17b1822093c930" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45883 } } peers { permanent_uuid: "14a70e196655445489a2f66d1d052f5f" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37349 } } peers { permanent_uuid: "781bfab4f65e45d6b7294d72e5e8db92" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 37951 } }
I20260620 18:43:07.187705 30874 raft_consensus.cc:374] T 0fc2559edd5c4b67b87638ba22874850 P 14a70e196655445489a2f66d1d052f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:07.187752 30874 raft_consensus.cc:729] T 0fc2559edd5c4b67b87638ba22874850 P 14a70e196655445489a2f66d1d052f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14a70e196655445489a2f66d1d052f5f, State: Initialized, Role: FOLLOWER
I20260620 18:43:07.187669 30873 raft_consensus.cc:348] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06715a3399dd45d38a17b1822093c930" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45883 } } peers { permanent_uuid: "14a70e196655445489a2f66d1d052f5f" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37349 } } peers { permanent_uuid: "781bfab4f65e45d6b7294d72e5e8db92" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 37951 } }
I20260620 18:43:07.187866 30873 raft_consensus.cc:374] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:07.187906 30873 raft_consensus.cc:729] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781bfab4f65e45d6b7294d72e5e8db92, State: Initialized, Role: FOLLOWER
I20260620 18:43:07.187902 30874 consensus_queue.cc:260] T 0fc2559edd5c4b67b87638ba22874850 P 14a70e196655445489a2f66d1d052f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06715a3399dd45d38a17b1822093c930" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45883 } } peers { permanent_uuid: "14a70e196655445489a2f66d1d052f5f" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37349 } } peers { permanent_uuid: "781bfab4f65e45d6b7294d72e5e8db92" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 37951 } }
I20260620 18:43:07.188032 30873 consensus_queue.cc:260] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06715a3399dd45d38a17b1822093c930" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45883 } } peers { permanent_uuid: "14a70e196655445489a2f66d1d052f5f" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37349 } } peers { permanent_uuid: "781bfab4f65e45d6b7294d72e5e8db92" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 37951 } }
I20260620 18:43:07.188256 30874 ts_tablet_manager.cc:1428] T 0fc2559edd5c4b67b87638ba22874850 P 14a70e196655445489a2f66d1d052f5f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:43:07.188416 30588 heartbeater.cc:503] Master 127.10.250.126:38173 was elected leader, sending a full tablet report...
I20260620 18:43:07.188426 30856 heartbeater.cc:503] Master 127.10.250.126:38173 was elected leader, sending a full tablet report...
I20260620 18:43:07.188328 30873 ts_tablet_manager.cc:1428] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:43:07.189710 30875 raft_consensus.cc:348] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06715a3399dd45d38a17b1822093c930" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45883 } } peers { permanent_uuid: "14a70e196655445489a2f66d1d052f5f" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37349 } } peers { permanent_uuid: "781bfab4f65e45d6b7294d72e5e8db92" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 37951 } }
I20260620 18:43:07.189939 30875 raft_consensus.cc:374] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:07.189991 30875 raft_consensus.cc:729] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06715a3399dd45d38a17b1822093c930, State: Initialized, Role: FOLLOWER
I20260620 18:43:07.190116 30875 consensus_queue.cc:260] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06715a3399dd45d38a17b1822093c930" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45883 } } peers { permanent_uuid: "14a70e196655445489a2f66d1d052f5f" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37349 } } peers { permanent_uuid: "781bfab4f65e45d6b7294d72e5e8db92" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 37951 } }
I20260620 18:43:07.190445 30875 ts_tablet_manager.cc:1428] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:43:07.190500 30722 heartbeater.cc:503] Master 127.10.250.126:38173 was elected leader, sending a full tablet report...
W20260620 18:43:07.193351 30723 tablet.cc:2369] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:07.217727 30880 raft_consensus.cc:482] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:43:07.217943 30880 raft_consensus.cc:504] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06715a3399dd45d38a17b1822093c930" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45883 } } peers { permanent_uuid: "14a70e196655445489a2f66d1d052f5f" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37349 } } peers { permanent_uuid: "781bfab4f65e45d6b7294d72e5e8db92" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 37951 } }
I20260620 18:43:07.218346 30880 leader_election.cc:288] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06715a3399dd45d38a17b1822093c930 (127.10.250.66:45883), 14a70e196655445489a2f66d1d052f5f (127.10.250.65:37349)
I20260620 18:43:07.222786 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.218805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:36381 (local address 127.10.250.65:37349)
0620 18:43:07.218847 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.218849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.218973 (+ 124us) server_negotiation.cc:407] Connection header received
0620 18:43:07.219047 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.219048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.219056 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.219108 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:07.219267 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.219993 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.221617 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.222330 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.222419 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.222428 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.222675 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.222718 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:43:07.222786 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.218765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:52591 (local address 127.10.250.66:45883)
0620 18:43:07.218818 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.218820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.218865 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:43:07.218897 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.218900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.218915 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.218993 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:07.219217 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.219994 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.221617 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.222330 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.222419 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.222428 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.222664 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.222708 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 18:43:07.222798 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.218650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:45883 (local address 127.10.250.67:52591)
0620 18:43:07.218717 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.218732 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:07.218795 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:07.219028 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:07.219029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:07.219031 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:07.219125 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.219127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.220100 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.220102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.221491 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.221494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.222446 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.222447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.222569 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.222574 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:43:07.222642 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 18:43:07.222695 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":102}
I20260620 18:43:07.222807 30882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.218650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:37349 (local address 127.10.250.67:36381)
0620 18:43:07.218878 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.218906 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:07.218954 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:07.219126 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:07.219127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:07.219128 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:07.219194 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.219195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.220108 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.220108 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.221491 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.221494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.222446 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.222447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.222595 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.222599 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:43:07.222653 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 18:43:07.222695 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":67,"thread_start_us":132,"threads_started":1}
I20260620 18:43:07.223197 30676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc2559edd5c4b67b87638ba22874850" candidate_uuid: "781bfab4f65e45d6b7294d72e5e8db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06715a3399dd45d38a17b1822093c930" is_pre_election: true
I20260620 18:43:07.223222 30542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc2559edd5c4b67b87638ba22874850" candidate_uuid: "781bfab4f65e45d6b7294d72e5e8db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14a70e196655445489a2f66d1d052f5f" is_pre_election: true
I20260620 18:43:07.223387 30542 raft_consensus.cc:2454] T 0fc2559edd5c4b67b87638ba22874850 P 14a70e196655445489a2f66d1d052f5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 781bfab4f65e45d6b7294d72e5e8db92 in term 0.
I20260620 18:43:07.223387 30676 raft_consensus.cc:2454] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 781bfab4f65e45d6b7294d72e5e8db92 in term 0.
I20260620 18:43:07.223661 30744 leader_election.cc:302] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06715a3399dd45d38a17b1822093c930, 781bfab4f65e45d6b7294d72e5e8db92; no voters:
I20260620 18:43:07.223805 30880 raft_consensus.cc:2781] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:43:07.223897 30880 raft_consensus.cc:482] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:43:07.223935 30880 raft_consensus.cc:3037] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:07.224687 30880 raft_consensus.cc:504] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06715a3399dd45d38a17b1822093c930" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45883 } } peers { permanent_uuid: "14a70e196655445489a2f66d1d052f5f" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37349 } } peers { permanent_uuid: "781bfab4f65e45d6b7294d72e5e8db92" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 37951 } }
I20260620 18:43:07.224865 30880 leader_election.cc:288] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [CANDIDATE]: Term 1 election: Requested vote from peers 06715a3399dd45d38a17b1822093c930 (127.10.250.66:45883), 14a70e196655445489a2f66d1d052f5f (127.10.250.65:37349)
I20260620 18:43:07.225134 30676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc2559edd5c4b67b87638ba22874850" candidate_uuid: "781bfab4f65e45d6b7294d72e5e8db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06715a3399dd45d38a17b1822093c930"
I20260620 18:43:07.225172 30542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc2559edd5c4b67b87638ba22874850" candidate_uuid: "781bfab4f65e45d6b7294d72e5e8db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14a70e196655445489a2f66d1d052f5f"
I20260620 18:43:07.225216 30676 raft_consensus.cc:3037] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:07.225279 30542 raft_consensus.cc:3037] T 0fc2559edd5c4b67b87638ba22874850 P 14a70e196655445489a2f66d1d052f5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:07.226049 30542 raft_consensus.cc:2454] T 0fc2559edd5c4b67b87638ba22874850 P 14a70e196655445489a2f66d1d052f5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 781bfab4f65e45d6b7294d72e5e8db92 in term 1.
I20260620 18:43:07.226047 30676 raft_consensus.cc:2454] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 781bfab4f65e45d6b7294d72e5e8db92 in term 1.
I20260620 18:43:07.226261 30743 leader_election.cc:302] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14a70e196655445489a2f66d1d052f5f, 781bfab4f65e45d6b7294d72e5e8db92; no voters:
I20260620 18:43:07.226372 30880 raft_consensus.cc:2781] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:07.226560 30880 raft_consensus.cc:686] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [term 1 LEADER]: Becoming Leader. State: Replica: 781bfab4f65e45d6b7294d72e5e8db92, State: Running, Role: LEADER
I20260620 18:43:07.226672 30880 consensus_queue.cc:237] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06715a3399dd45d38a17b1822093c930" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45883 } } peers { permanent_uuid: "14a70e196655445489a2f66d1d052f5f" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37349 } } peers { permanent_uuid: "781bfab4f65e45d6b7294d72e5e8db92" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 37951 } }
I20260620 18:43:07.227665 30396 catalog_manager.cc:5495] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 781bfab4f65e45d6b7294d72e5e8db92 (127.10.250.67). New cstate: current_term: 1 leader_uuid: "781bfab4f65e45d6b7294d72e5e8db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06715a3399dd45d38a17b1822093c930" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14a70e196655445489a2f66d1d052f5f" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "781bfab4f65e45d6b7294d72e5e8db92" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 37951 } health_report { overall_health: HEALTHY } } }
I20260620 18:43:07.237550 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.234882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.10.250.67:37951)
0620 18:43:07.234944 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.234946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.234958 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:43:07.235030 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.235031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.235050 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.235135 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:07.235296 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.235798 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.236593 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.236821 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.236907 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.237088 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:07.237200 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:07.237359 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.237429 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.237453 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:43:07.239576 30542 raft_consensus.cc:1261] T 0fc2559edd5c4b67b87638ba22874850 P 14a70e196655445489a2f66d1d052f5f [term 1 FOLLOWER]: Refusing update from remote peer 781bfab4f65e45d6b7294d72e5e8db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:07.239578 30676 raft_consensus.cc:1261] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930 [term 1 FOLLOWER]: Refusing update from remote peer 781bfab4f65e45d6b7294d72e5e8db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:07.239969 30880 consensus_queue.cc:1035] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a70e196655445489a2f66d1d052f5f" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 37349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:07.240195 30883 consensus_queue.cc:1035] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06715a3399dd45d38a17b1822093c930" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 45883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:07.243499 30888 mvcc.cc:205] Tried to move back new op lower bound from 7298994123730509824 to 7298994123681296384. Current Snapshot: MvccSnapshot[applied={T|T < 7298994123730509824}]
I20260620 18:43:07.245805 30396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:07.245920 30396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:07.246810 30396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05cb6b9a858c4bb09ce393f23aa4bde1: Sending DeleteTablet for 3 replicas of tablet 0fc2559edd5c4b67b87638ba22874850
I20260620 18:43:07.247248 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fc2559edd5c4b67b87638ba22874850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:07 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:60110
I20260620 18:43:07.247264 30522 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fc2559edd5c4b67b87638ba22874850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:07 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:56322
I20260620 18:43:07.247253 30656 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fc2559edd5c4b67b87638ba22874850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:07 UTC) from {username='kudu', principal='kudu/127.10.250.126@KRBTEST.COM'} at 127.0.0.1:43716
I20260620 18:43:07.247516 30898 tablet_replica.cc:330] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92: stopping tablet replica
I20260620 18:43:07.247524 30897 tablet_replica.cc:330] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930: stopping tablet replica
I20260620 18:43:07.247622 30898 raft_consensus.cc:2229] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:43:07.247644 30897 raft_consensus.cc:2229] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:07.247745 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 30459
I20260620 18:43:07.247874 30897 raft_consensus.cc:2258] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:07.249238 30897 ts_tablet_manager.cc:1905] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:07.247805 30898 raft_consensus.cc:2258] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:07.249861 30898 ts_tablet_manager.cc:1905] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:07.251163 30897 ts_tablet_manager.cc:1918] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:07.251273 30897 log.cc:1196] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/0fc2559edd5c4b67b87638ba22874850
I20260620 18:43:07.251555 30897 ts_tablet_manager.cc:1939] T 0fc2559edd5c4b67b87638ba22874850 P 06715a3399dd45d38a17b1822093c930: Deleting consensus metadata
I20260620 18:43:07.251904 30898 ts_tablet_manager.cc:1918] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:07.251987 30898 log.cc:1196] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/0fc2559edd5c4b67b87638ba22874850
I20260620 18:43:07.252066 30382 catalog_manager.cc:4841] TS 06715a3399dd45d38a17b1822093c930 (127.10.250.66:45883): tablet 0fc2559edd5c4b67b87638ba22874850 (table test-table [id=9f6c10fa12e24787817642099ab6c6de]) successfully deleted
I20260620 18:43:07.252256 30898 ts_tablet_manager.cc:1939] T 0fc2559edd5c4b67b87638ba22874850 P 781bfab4f65e45d6b7294d72e5e8db92: Deleting consensus metadata
I20260620 18:43:07.252676 30383 catalog_manager.cc:4841] TS 781bfab4f65e45d6b7294d72e5e8db92 (127.10.250.67:37951): tablet 0fc2559edd5c4b67b87638ba22874850 (table test-table [id=9f6c10fa12e24787817642099ab6c6de]) successfully deleted
W20260620 18:43:07.253674 30381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:37349 (error 108)
I20260620 18:43:07.254146 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.254049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:37349 (local address 127.0.0.1:56334)
0620 18:43:07.254085 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.254116 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:37349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260620 18:43:07.254307 30381 catalog_manager.cc:4584] TS 14a70e196655445489a2f66d1d052f5f (127.10.250.65:37349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0fc2559edd5c4b67b87638ba22874850: Network error: Client connection negotiation failed: client connection to 127.10.250.65:37349: connect: Connection refused (error 111)
I20260620 18:43:07.255205 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 30593
I20260620 18:43:07.260886 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 30727
I20260620 18:43:07.265602 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 30365
2026-06-20T18:43:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2018 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T18:43:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:07Z Disabled control of system clock
I20260620 18:43:07.294270 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:37067
--webserver_interface=127.10.250.126
--webserver_port=0
--builtin_ntp_servers=127.10.250.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:37067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:43:07.380388 30903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:07.380640 30903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:07.380662 30903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:07.381847 30903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:07.381887 30903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:07.381901 30903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:07.381914 30903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:07.381927 30903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:07.383287 30903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:37067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:37067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:07.383502 30903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:07.383759 30903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:07.389379 30911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:07.389392 30903 server_base.cc:1034] running on GCE node
W20260620 18:43:07.389369 30909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:07.389369 30908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:07.389959 30903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:07.390293 30903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:07.391472 30903 hybrid_clock.cc:648] HybridClock initialized: now 1781980987391448 us; error 47 us; skew 500 ppm
I20260620 18:43:07.392809 30903 webserver.cc:460] Webserver started at http://127.10.250.126:39425/ using document root <none> and password file <none>
I20260620 18:43:07.393103 30903 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:07.393175 30903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:07.393339 30903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:07.394485 30903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "67ddeff7a33c46bab26eb422f7c51968"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-wp6g"
server_key: "626dd8527fee16c2e4f28234931a3cba"
server_key_iv: "2cd84c9185ca0bb9c34b3de8162769d0"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.394942 30903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "67ddeff7a33c46bab26eb422f7c51968"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-wp6g"
server_key: "626dd8527fee16c2e4f28234931a3cba"
server_key_iv: "2cd84c9185ca0bb9c34b3de8162769d0"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.396385 30903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:43:07.397323 30917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.397579 30903 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:07.397699 30903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "67ddeff7a33c46bab26eb422f7c51968"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-wp6g"
server_key: "626dd8527fee16c2e4f28234931a3cba"
server_key_iv: "2cd84c9185ca0bb9c34b3de8162769d0"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.397784 30903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:07.407114 30903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:07.407469 30903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:07.407630 30903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:07.412019 30903 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:37067
I20260620 18:43:07.412083 30969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:37067 every 8 connection(s)
I20260620 18:43:07.412472 30903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:07.413652 30970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:07.416251 30970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968: Bootstrap starting.
I20260620 18:43:07.416885 30970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:07.417220 30970 log.cc:824] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:07.418004 30970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968: No bootstrap required, opened a new log
I20260620 18:43:07.419063 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 30903
I20260620 18:43:07.419289 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:07.419474 11241 external_mini_cluster.cc:1428] Setting key 4847f27855c43ce8ced8a81eb9301690
I20260620 18:43:07.419960 30970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ddeff7a33c46bab26eb422f7c51968" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37067 } }
I20260620 18:43:07.420162 30970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:07.420202 30970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67ddeff7a33c46bab26eb422f7c51968, State: Initialized, Role: FOLLOWER
I20260620 18:43:07.420346 30970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ddeff7a33c46bab26eb422f7c51968" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37067 } }
I20260620 18:43:07.420435 30970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:07.420466 30970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:07.420516 30970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:07.421279 30970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ddeff7a33c46bab26eb422f7c51968" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37067 } }
I20260620 18:43:07.421415 30970 leader_election.cc:302] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67ddeff7a33c46bab26eb422f7c51968; no voters:
I20260620 18:43:07.421610 30970 leader_election.cc:288] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:07.421747 30975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:07.421871 30970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:07.421888 30975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [term 1 LEADER]: Becoming Leader. State: Replica: 67ddeff7a33c46bab26eb422f7c51968, State: Running, Role: LEADER
I20260620 18:43:07.421986 30975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ddeff7a33c46bab26eb422f7c51968" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37067 } }
I20260620 18:43:07.422344 30975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67ddeff7a33c46bab26eb422f7c51968. Latest consensus state: current_term: 1 leader_uuid: "67ddeff7a33c46bab26eb422f7c51968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ddeff7a33c46bab26eb422f7c51968" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37067 } } }
I20260620 18:43:07.422413 30975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:07.422689 30977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67ddeff7a33c46bab26eb422f7c51968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ddeff7a33c46bab26eb422f7c51968" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 37067 } } }
I20260620 18:43:07.422780 30977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:07.422925 30981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:07.423544 30981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:07.425354 30981 catalog_manager.cc:1347] Generated new cluster ID: aab60da7d8f14805a8337b8fb513d836
I20260620 18:43:07.425421 30981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:07.427314 30974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.420071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.10.250.126:37067)
0620 18:43:07.420317 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.420327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.420360 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:43:07.420480 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.420487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.420561 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.420719 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:07.420998 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.423662 (+ 2664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.425012 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.425691 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.425833 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.426604 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:07.426608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:07.426609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:07.426812 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:07.426863 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:07.426875 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:07.426961 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.427040 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.427125 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":90,"threads_started":1}
I20260620 18:43:07.452064 30981 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:07.452858 30981 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:07.458681 30981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968: Generated new TSK 0
I20260620 18:43:07.458925 30981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:07.463306 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.65:0
--local_ip_for_outbound_sockets=127.10.250.65
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37067
--builtin_ntp_servers=127.10.250.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:43:07.550395 30994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:07.550700 30994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:07.550745 30994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:07.552026 30994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:07.552101 30994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:07.552148 30994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.65
I20260620 18:43:07.553717 30994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.250.65
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.65
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:07.554003 30994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:07.554318 30994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:07.560851 31000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:07.560851 31002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:07.561058 30999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:07.560926 30994 server_base.cc:1034] running on GCE node
I20260620 18:43:07.561369 30994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:07.561640 30994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:07.562898 30994 hybrid_clock.cc:648] HybridClock initialized: now 1781980987562873 us; error 41 us; skew 500 ppm
I20260620 18:43:07.564373 30994 webserver.cc:460] Webserver started at http://127.10.250.65:32967/ using document root <none> and password file <none>
I20260620 18:43:07.564682 30994 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:07.564761 30994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:07.564926 30994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:07.566082 30994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/data/instance:
uuid: "5a4922f55ec8493ea49bd598f5124260"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-wp6g"
server_key: "a072d5b11152c5b806021019e9bb33d6"
server_key_iv: "1bfcd31704beb75698e8de1428aee04d"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.566475 30994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance:
uuid: "5a4922f55ec8493ea49bd598f5124260"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-wp6g"
server_key: "a072d5b11152c5b806021019e9bb33d6"
server_key_iv: "1bfcd31704beb75698e8de1428aee04d"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.568017 30994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:07.568905 31008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.569200 30994 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:07.569324 30994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/wal
uuid: "5a4922f55ec8493ea49bd598f5124260"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-wp6g"
server_key: "a072d5b11152c5b806021019e9bb33d6"
server_key_iv: "1bfcd31704beb75698e8de1428aee04d"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.569432 30994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:07.580754 30994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:07.581274 30994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:07.581482 30994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:07.581773 30994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:07.582176 30994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:07.582237 30994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.582280 30994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:07.582305 30994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.588567 30994 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.65:34845
I20260620 18:43:07.588645 31121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.65:34845 every 8 connection(s)
I20260620 18:43:07.589073 30994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/data/info.pb
I20260620 18:43:07.589511 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 30994
I20260620 18:43:07.589609 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-0/wal/instance
I20260620 18:43:07.589810 11241 external_mini_cluster.cc:1428] Setting key 8a58ff9b3b78ef922c283a33c39119fc
I20260620 18:43:07.591097 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.66:0
--local_ip_for_outbound_sockets=127.10.250.66
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37067
--builtin_ntp_servers=127.10.250.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:43:07.595624 30974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.590559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.65:58173 (local address 127.10.250.126:37067)
0620 18:43:07.590607 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.590610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.590987 (+ 377us) server_negotiation.cc:407] Connection header received
0620 18:43:07.591616 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.591618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.591634 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.591711 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:07.592416 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.592936 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.594017 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.594235 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.594324 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.595315 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:07.595316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:07.595317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:07.595324 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:07.595340 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:07.595345 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:07.595415 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.595499 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.595532 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:43:07.595897 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.590609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:37067 (local address 127.10.250.65:58173)
0620 18:43:07.590863 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.590900 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:07.591502 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:07.591772 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:07.591775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:07.591784 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:07.592309 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.592314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.593040 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.593041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.593904 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.593907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.594340 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.594341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.594486 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.595153 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:07.595154 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:07.595185 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:07.595212 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:07.595417 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:07.595424 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:07.595427 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:43:07.595494 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 18:43:07.595605 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":114,"threads_started":1}
I20260620 18:43:07.596428 31122 heartbeater.cc:347] Connected to a master server at 127.10.250.126:37067
I20260620 18:43:07.596567 31122 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:07.596808 31122 heartbeater.cc:511] Master 127.10.250.126:37067 requested a full tablet report, sending...
I20260620 18:43:07.597522 30934 ts_manager.cc:194] Registered new tserver with Master: 5a4922f55ec8493ea49bd598f5124260 (127.10.250.65:34845)
I20260620 18:43:07.598598 30934 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.250.65:58173
W20260620 18:43:07.677580 31125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:07.677832 31125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:07.677858 31125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:07.679169 31125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:07.679217 31125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:07.679245 31125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.66
I20260620 18:43:07.680652 31125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.250.66
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.66
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:07.680881 31125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:07.681166 31125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:07.686728 31131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:07.686722 31133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:07.686792 31125 server_base.cc:1034] running on GCE node
W20260620 18:43:07.686744 31130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:07.687276 31125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:07.687500 31125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:07.688663 31125 hybrid_clock.cc:648] HybridClock initialized: now 1781980987688644 us; error 35 us; skew 500 ppm
I20260620 18:43:07.690073 31125 webserver.cc:460] Webserver started at http://127.10.250.66:44409/ using document root <none> and password file <none>
I20260620 18:43:07.690320 31125 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:07.690366 31125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:07.690481 31125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:07.691676 31125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/data/instance:
uuid: "0cc3a8c8f94344569fdb80206ada6640"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-wp6g"
server_key: "e9207c5252805c3bb326a316b00a82e2"
server_key_iv: "3a65a1131da3ee10524393de8a0a5fc4"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.691993 31125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance:
uuid: "0cc3a8c8f94344569fdb80206ada6640"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-wp6g"
server_key: "e9207c5252805c3bb326a316b00a82e2"
server_key_iv: "3a65a1131da3ee10524393de8a0a5fc4"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.693312 31125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 18:43:07.694110 31139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.694316 31125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:07.694435 31125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/wal
uuid: "0cc3a8c8f94344569fdb80206ada6640"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-wp6g"
server_key: "e9207c5252805c3bb326a316b00a82e2"
server_key_iv: "3a65a1131da3ee10524393de8a0a5fc4"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.694518 31125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:07.722600 31125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:07.722952 31125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:07.723127 31125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:07.723377 31125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:07.723744 31125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:07.723798 31125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.723834 31125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:07.723861 31125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.729749 31125 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.66:41903
I20260620 18:43:07.729799 31252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.66:41903 every 8 connection(s)
I20260620 18:43:07.730147 31125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/data/info.pb
I20260620 18:43:07.735826 30974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.730925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.66:58633 (local address 127.10.250.126:37067)
0620 18:43:07.730971 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.730973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.731281 (+ 308us) server_negotiation.cc:407] Connection header received
0620 18:43:07.731888 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.731889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.731905 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.731960 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:07.732683 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.733188 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.734239 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.734438 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.734516 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.735489 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:07.735491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:07.735491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:07.735496 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:07.735515 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:07.735524 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:07.735586 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.735703 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.735741 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:43:07.736061 31255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.730902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:37067 (local address 127.10.250.66:58633)
0620 18:43:07.731129 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.731185 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:07.731772 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:07.732048 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:07.732052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:07.732069 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:07.732586 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.732590 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.733293 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.733295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.734117 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.734119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.734585 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.734586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.734721 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.735322 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:07.735323 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:07.735354 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:07.735378 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:07.735602 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:07.735609 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:07.735626 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 18:43:07.735702 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 18:43:07.735818 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260620 18:43:07.736747 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 31125
I20260620 18:43:07.736830 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/wal/instance
I20260620 18:43:07.736891 31253 heartbeater.cc:347] Connected to a master server at 127.10.250.126:37067
I20260620 18:43:07.736990 31253 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:07.737058 11241 external_mini_cluster.cc:1428] Setting key c30a567878aa7611990c893c9a20a8c8
I20260620 18:43:07.737211 31253 heartbeater.cc:511] Master 127.10.250.126:37067 requested a full tablet report, sending...
I20260620 18:43:07.737764 30933 ts_manager.cc:194] Registered new tserver with Master: 0cc3a8c8f94344569fdb80206ada6640 (127.10.250.66:41903)
I20260620 18:43:07.738226 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.250.67:0
--local_ip_for_outbound_sockets=127.10.250.67
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37067
--builtin_ntp_servers=127.10.250.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:43:07.738318 30933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.250.66:58633
W20260620 18:43:07.828409 31256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:07.828636 31256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:07.828661 31256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:07.829900 31256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:07.829941 31256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:07.829969 31256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.250.67
I20260620 18:43:07.831699 31256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.250.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.250.67
--webserver_port=0
--tserver_master_addrs=127.10.250.126:37067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.250.67
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:07.831995 31256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:07.832333 31256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:07.837798 31261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:07.837857 31256 server_base.cc:1034] running on GCE node
W20260620 18:43:07.837810 31264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:07.837954 31262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:07.838203 31256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:07.838549 31256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:07.839776 31256 hybrid_clock.cc:648] HybridClock initialized: now 1781980987839755 us; error 30 us; skew 500 ppm
I20260620 18:43:07.841226 31256 webserver.cc:460] Webserver started at http://127.10.250.67:37015/ using document root <none> and password file <none>
I20260620 18:43:07.841498 31256 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:07.841547 31256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:07.841719 31256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:07.842929 31256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/data/instance:
uuid: "33ff019bd2a14188a6de510fc848790c"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-wp6g"
server_key: "a17600a6843be8c2540de4ea0d392715"
server_key_iv: "b26377292a240c026f47151400dd6302"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.843299 31256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance:
uuid: "33ff019bd2a14188a6de510fc848790c"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-wp6g"
server_key: "a17600a6843be8c2540de4ea0d392715"
server_key_iv: "b26377292a240c026f47151400dd6302"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.844739 31256 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:43:07.845638 31270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.845860 31256 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:07.845971 31256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/wal
uuid: "33ff019bd2a14188a6de510fc848790c"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-wp6g"
server_key: "a17600a6843be8c2540de4ea0d392715"
server_key_iv: "b26377292a240c026f47151400dd6302"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.846053 31256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:07.863792 31256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:07.864166 31256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:07.864333 31256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:07.864598 31256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:07.864959 31256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:07.865015 31256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.865051 31256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:07.865080 31256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.871018 31256 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.67:41447
I20260620 18:43:07.871101 31383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.67:41447 every 8 connection(s)
I20260620 18:43:07.871445 31256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/data/info.pb
I20260620 18:43:07.876822 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 31256
I20260620 18:43:07.876914 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/wal/instance
I20260620 18:43:07.877221 11241 external_mini_cluster.cc:1428] Setting key 8b5c2a8cae11c2e87e27cec027130d3f
I20260620 18:43:07.877367 30974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.872303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:37063 (local address 127.10.250.126:37067)
0620 18:43:07.872390 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.872393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.872687 (+ 294us) server_negotiation.cc:407] Connection header received
0620 18:43:07.873316 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.873317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.873329 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.873374 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:07.874103 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.874632 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.875685 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.875881 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.875954 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.876994 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:07.876996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:07.876997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:07.877004 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:07.877025 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:07.877030 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:07.877109 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.877247 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.877283 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:43:07.877702 31386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.872293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:37067 (local address 127.10.250.67:37063)
0620 18:43:07.872522 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.872557 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:07.873194 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:07.873451 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:07.873455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:07.873463 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:07.873976 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.873980 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.874761 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.874762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.875579 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.875581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.875975 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.875976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.876117 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.876783 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:07.876783 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:07.876815 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:07.876849 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:07.877137 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:07.877143 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:07.877146 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:43:07.877224 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:43:07.877380 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":87,"threads_started":1}
I20260620 18:43:07.878159 31384 heartbeater.cc:347] Connected to a master server at 127.10.250.126:37067
I20260620 18:43:07.878278 31384 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:07.878484 31384 heartbeater.cc:511] Master 127.10.250.126:37067 requested a full tablet report, sending...
I20260620 18:43:07.878978 30933 ts_manager.cc:194] Registered new tserver with Master: 33ff019bd2a14188a6de510fc848790c (127.10.250.67:41447)
I20260620 18:43:07.879418 30933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.250.67:37063
I20260620 18:43:07.889576 11241 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:43:07.895183 30974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.891307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.10.250.126:37067)
0620 18:43:07.891403 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.891405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.891504 (+ 99us) server_negotiation.cc:407] Connection header received
0620 18:43:07.891681 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.891683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.891699 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.891767 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:07.891989 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.892513 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.893730 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.893939 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.894032 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.894882 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:07.894884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:07.894884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:07.894890 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:07.894909 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:07.894913 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:07.894972 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.895083 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.895116 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:43:07.897122 30933 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:43:07.899021 30933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:43:07.906198 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.902060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40836 (local address 127.10.250.65:34845)
0620 18:43:07.902224 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.902230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.902282 (+ 52us) server_negotiation.cc:407] Connection header received
0620 18:43:07.902393 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.902395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.902408 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.902571 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:07.902774 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.903996 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.905216 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.905726 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.905820 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.905913 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.906040 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.906092 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260620 18:43:07.906186 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.901780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:34845 (local address 127.0.0.1:40836)
0620 18:43:07.902219 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.902231 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:07.902295 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:07.902644 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:07.902645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:07.902647 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:07.902693 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.902695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.904096 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.904097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.905105 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.905107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.905820 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.905821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.905905 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.905920 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 18:43:07.906021 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:43:07.906096 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":22,"thread_start_us":58,"threads_started":1}
I20260620 18:43:07.906231 31394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.901796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55354 (local address 127.10.250.66:41903)
0620 18:43:07.901988 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.901993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.902107 (+ 114us) server_negotiation.cc:407] Connection header received
0620 18:43:07.902269 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.902271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.902289 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.902365 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:07.902672 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.903485 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.904913 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.905681 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.905751 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.905860 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.906076 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.906121 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
I20260620 18:43:07.906431 31392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.901716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:41903 (local address 127.0.0.1:55354)
0620 18:43:07.901978 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.902023 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:07.902129 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:07.902394 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:07.902398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:07.902400 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:07.902492 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.902495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.903588 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.903590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.904814 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.904817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.905900 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.905901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.905963 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.905967 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:43:07.906024 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 18:43:07.906337 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":254,"thread_start_us":95,"threads_started":1}
I20260620 18:43:07.906563 31395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.901780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.67:41447 (local address 127.0.0.1:39234)
0620 18:43:07.902416 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.902425 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:07.902468 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:07.902757 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:07.902758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:07.902760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:07.902805 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.902806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.904106 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.904107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.905167 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:07.905170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.906199 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.906200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:07.906266 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.906270 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:43:07.906414 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 18:43:07.906462 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":13,"thread_start_us":87,"threads_started":1}
I20260620 18:43:07.906600 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.902026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39234 (local address 127.10.250.67:41447)
0620 18:43:07.902229 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.902235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.902548 (+ 313us) server_negotiation.cc:407] Connection header received
0620 18:43:07.902593 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.902596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.902612 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.902686 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:07.902891 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.904013 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.905270 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.906018 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.906076 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.906165 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.906434 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.906474 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
I20260620 18:43:07.907501 31187 tablet_service.cc:1467] Processing CreateTablet for tablet 9465d5c178f24156a4da347bda108403 (DEFAULT_TABLE table=test-table [id=c50d3926c3a6438eb9f44216902ce9f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:07.907881 31187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9465d5c178f24156a4da347bda108403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:07.907665 31056 tablet_service.cc:1467] Processing CreateTablet for tablet 9465d5c178f24156a4da347bda108403 (DEFAULT_TABLE table=test-table [id=c50d3926c3a6438eb9f44216902ce9f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:07.908178 31056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9465d5c178f24156a4da347bda108403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:07.911084 31318 tablet_service.cc:1467] Processing CreateTablet for tablet 9465d5c178f24156a4da347bda108403 (DEFAULT_TABLE table=test-table [id=c50d3926c3a6438eb9f44216902ce9f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:07.911471 31398 tablet_bootstrap.cc:492] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640: Bootstrap starting.
I20260620 18:43:07.911496 31318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9465d5c178f24156a4da347bda108403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:07.911988 31399 tablet_bootstrap.cc:492] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260: Bootstrap starting.
I20260620 18:43:07.912230 31398 tablet_bootstrap.cc:654] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:07.912533 31398 log.cc:824] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:07.912798 31399 tablet_bootstrap.cc:654] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:07.913158 31399 log.cc:824] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:07.913395 31398 tablet_bootstrap.cc:492] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640: No bootstrap required, opened a new log
I20260620 18:43:07.913482 31398 ts_tablet_manager.cc:1397] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:43:07.914211 31399 tablet_bootstrap.cc:492] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260: No bootstrap required, opened a new log
I20260620 18:43:07.914358 31399 ts_tablet_manager.cc:1397] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:43:07.916059 31402 tablet_bootstrap.cc:492] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c: Bootstrap starting.
I20260620 18:43:07.916217 31398 raft_consensus.cc:348] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc3a8c8f94344569fdb80206ada6640" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 41903 } } peers { permanent_uuid: "33ff019bd2a14188a6de510fc848790c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 41447 } } peers { permanent_uuid: "5a4922f55ec8493ea49bd598f5124260" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 34845 } }
I20260620 18:43:07.916533 31398 raft_consensus.cc:374] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:07.916595 31398 raft_consensus.cc:729] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc3a8c8f94344569fdb80206ada6640, State: Initialized, Role: FOLLOWER
I20260620 18:43:07.916783 31398 consensus_queue.cc:260] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc3a8c8f94344569fdb80206ada6640" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 41903 } } peers { permanent_uuid: "33ff019bd2a14188a6de510fc848790c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 41447 } } peers { permanent_uuid: "5a4922f55ec8493ea49bd598f5124260" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 34845 } }
I20260620 18:43:07.916945 31402 tablet_bootstrap.cc:654] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:07.917096 31398 ts_tablet_manager.cc:1428] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260620 18:43:07.916988 31399 raft_consensus.cc:348] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc3a8c8f94344569fdb80206ada6640" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 41903 } } peers { permanent_uuid: "33ff019bd2a14188a6de510fc848790c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 41447 } } peers { permanent_uuid: "5a4922f55ec8493ea49bd598f5124260" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 34845 } }
I20260620 18:43:07.917173 31399 raft_consensus.cc:374] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:07.917222 31399 raft_consensus.cc:729] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4922f55ec8493ea49bd598f5124260, State: Initialized, Role: FOLLOWER
I20260620 18:43:07.917353 31253 heartbeater.cc:503] Master 127.10.250.126:37067 was elected leader, sending a full tablet report...
I20260620 18:43:07.917340 31402 log.cc:824] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:07.917335 31399 consensus_queue.cc:260] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc3a8c8f94344569fdb80206ada6640" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 41903 } } peers { permanent_uuid: "33ff019bd2a14188a6de510fc848790c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 41447 } } peers { permanent_uuid: "5a4922f55ec8493ea49bd598f5124260" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 34845 } }
I20260620 18:43:07.917608 31399 ts_tablet_manager.cc:1428] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 18:43:07.917672 31122 heartbeater.cc:503] Master 127.10.250.126:37067 was elected leader, sending a full tablet report...
I20260620 18:43:07.918363 31402 tablet_bootstrap.cc:492] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c: No bootstrap required, opened a new log
I20260620 18:43:07.918442 31402 ts_tablet_manager.cc:1397] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:43:07.920682 31402 raft_consensus.cc:348] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc3a8c8f94344569fdb80206ada6640" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 41903 } } peers { permanent_uuid: "33ff019bd2a14188a6de510fc848790c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 41447 } } peers { permanent_uuid: "5a4922f55ec8493ea49bd598f5124260" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 34845 } }
I20260620 18:43:07.920866 31402 raft_consensus.cc:374] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:07.920889 31402 raft_consensus.cc:729] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ff019bd2a14188a6de510fc848790c, State: Initialized, Role: FOLLOWER
I20260620 18:43:07.921049 31402 consensus_queue.cc:260] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc3a8c8f94344569fdb80206ada6640" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 41903 } } peers { permanent_uuid: "33ff019bd2a14188a6de510fc848790c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 41447 } } peers { permanent_uuid: "5a4922f55ec8493ea49bd598f5124260" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 34845 } }
I20260620 18:43:07.921363 31402 ts_tablet_manager.cc:1428] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 18:43:07.921512 31384 heartbeater.cc:503] Master 127.10.250.126:37067 was elected leader, sending a full tablet report...
W20260620 18:43:07.980822 31254 tablet.cc:2369] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:43:08.090678 31123 tablet.cc:2369] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:43:08.122095 31385 tablet.cc:2369] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:08.245261 31406 raft_consensus.cc:482] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:43:08.245404 31406 raft_consensus.cc:504] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc3a8c8f94344569fdb80206ada6640" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 41903 } } peers { permanent_uuid: "33ff019bd2a14188a6de510fc848790c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 41447 } } peers { permanent_uuid: "5a4922f55ec8493ea49bd598f5124260" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 34845 } }
I20260620 18:43:08.245776 31406 leader_election.cc:288] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cc3a8c8f94344569fdb80206ada6640 (127.10.250.66:41903), 5a4922f55ec8493ea49bd598f5124260 (127.10.250.65:34845)
I20260620 18:43:08.248881 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:08.246092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:57693 (local address 127.10.250.65:34845)
0620 18:43:08.246140 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:08.246142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:08.246150 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:43:08.246179 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:08.246180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:08.246192 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:08.246252 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:08.246479 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.247067 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.248151 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.248518 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.248601 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:08.248612 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:08.248763 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 18:43:08.248805 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 18:43:08.248881 31386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:08.245968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:34845 (local address 127.10.250.67:57693)
0620 18:43:08.246031 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:08.246043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:08.246096 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:08.246297 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:08.246299 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:08.246301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:08.246379 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:08.246381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.247146 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.247147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:08.248058 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:08.248061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.248620 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.248621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:08.248688 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:08.248692 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:43:08.248758 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 18:43:08.248804 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 18:43:08.249022 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:08.245998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.66:41903 (local address 127.10.250.67:53477)
0620 18:43:08.246211 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:08.246220 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:08.246269 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:08.246477 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:08.246478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:08.246481 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:08.246587 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:08.246590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.247236 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.247237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:08.248155 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:08.248158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.248743 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.248744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:08.248824 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:08.248827 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:43:08.248901 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 18:43:08.248939 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":141,"threads_started":1}
I20260620 18:43:08.249017 31394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:08.246102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.250.67:53477 (local address 127.10.250.66:41903)
0620 18:43:08.246150 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:08.246152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:08.246289 (+ 137us) server_negotiation.cc:407] Connection header received
0620 18:43:08.246320 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:08.246320 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:08.246331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:08.246403 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:08.246679 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.247156 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.248243 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.248653 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.248742 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:08.248749 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:08.248911 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 18:43:08.248956 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:43:08.249269 31076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9465d5c178f24156a4da347bda108403" candidate_uuid: "33ff019bd2a14188a6de510fc848790c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4922f55ec8493ea49bd598f5124260" is_pre_election: true
I20260620 18:43:08.249442 31076 raft_consensus.cc:2454] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33ff019bd2a14188a6de510fc848790c in term 0.
I20260620 18:43:08.249432 31207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9465d5c178f24156a4da347bda108403" candidate_uuid: "33ff019bd2a14188a6de510fc848790c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cc3a8c8f94344569fdb80206ada6640" is_pre_election: true
I20260620 18:43:08.249590 31207 raft_consensus.cc:2454] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33ff019bd2a14188a6de510fc848790c in term 0.
I20260620 18:43:08.249747 31272 leader_election.cc:302] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33ff019bd2a14188a6de510fc848790c, 5a4922f55ec8493ea49bd598f5124260; no voters:
I20260620 18:43:08.249898 31406 raft_consensus.cc:2781] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:43:08.249961 31406 raft_consensus.cc:482] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:43:08.249992 31406 raft_consensus.cc:3037] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:08.250761 31406 raft_consensus.cc:504] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc3a8c8f94344569fdb80206ada6640" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 41903 } } peers { permanent_uuid: "33ff019bd2a14188a6de510fc848790c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 41447 } } peers { permanent_uuid: "5a4922f55ec8493ea49bd598f5124260" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 34845 } }
I20260620 18:43:08.250914 31406 leader_election.cc:288] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [CANDIDATE]: Term 1 election: Requested vote from peers 0cc3a8c8f94344569fdb80206ada6640 (127.10.250.66:41903), 5a4922f55ec8493ea49bd598f5124260 (127.10.250.65:34845)
I20260620 18:43:08.251078 31076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9465d5c178f24156a4da347bda108403" candidate_uuid: "33ff019bd2a14188a6de510fc848790c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4922f55ec8493ea49bd598f5124260"
I20260620 18:43:08.251112 31207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9465d5c178f24156a4da347bda108403" candidate_uuid: "33ff019bd2a14188a6de510fc848790c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cc3a8c8f94344569fdb80206ada6640"
I20260620 18:43:08.251200 31076 raft_consensus.cc:3037] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:08.251196 31207 raft_consensus.cc:3037] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:08.251922 31207 raft_consensus.cc:2454] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33ff019bd2a14188a6de510fc848790c in term 1.
I20260620 18:43:08.252117 31273 leader_election.cc:302] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cc3a8c8f94344569fdb80206ada6640, 33ff019bd2a14188a6de510fc848790c; no voters:
I20260620 18:43:08.252166 31076 raft_consensus.cc:2454] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33ff019bd2a14188a6de510fc848790c in term 1.
I20260620 18:43:08.252245 31406 raft_consensus.cc:2781] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:08.252439 31406 raft_consensus.cc:686] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [term 1 LEADER]: Becoming Leader. State: Replica: 33ff019bd2a14188a6de510fc848790c, State: Running, Role: LEADER
I20260620 18:43:08.252555 31406 consensus_queue.cc:237] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc3a8c8f94344569fdb80206ada6640" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 41903 } } peers { permanent_uuid: "33ff019bd2a14188a6de510fc848790c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 41447 } } peers { permanent_uuid: "5a4922f55ec8493ea49bd598f5124260" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 34845 } }
I20260620 18:43:08.253499 30934 catalog_manager.cc:5495] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c reported cstate change: term changed from 0 to 1, leader changed from <none> to 33ff019bd2a14188a6de510fc848790c (127.10.250.67). New cstate: current_term: 1 leader_uuid: "33ff019bd2a14188a6de510fc848790c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc3a8c8f94344569fdb80206ada6640" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 41903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33ff019bd2a14188a6de510fc848790c" member_type: VOTER last_known_addr { host: "127.10.250.67" port: 41447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4922f55ec8493ea49bd598f5124260" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 34845 } health_report { overall_health: UNKNOWN } } }
I20260620 18:43:08.264341 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:08.261758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39240 (local address 127.10.250.67:41447)
0620 18:43:08.261824 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:08.261826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:08.261835 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:43:08.261906 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:08.261907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:08.261933 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:08.262014 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:08.262156 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.262703 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.263514 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.263720 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.263796 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:08.263969 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:08.264082 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:08.264180 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:08.264199 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 18:43:08.264219 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:43:08.266366 31076 raft_consensus.cc:1261] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260 [term 1 FOLLOWER]: Refusing update from remote peer 33ff019bd2a14188a6de510fc848790c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:08.266371 31207 raft_consensus.cc:1261] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640 [term 1 FOLLOWER]: Refusing update from remote peer 33ff019bd2a14188a6de510fc848790c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:08.266727 31406 consensus_queue.cc:1035] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cc3a8c8f94344569fdb80206ada6640" member_type: VOTER last_known_addr { host: "127.10.250.66" port: 41903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:08.266881 31408 consensus_queue.cc:1035] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4922f55ec8493ea49bd598f5124260" member_type: VOTER last_known_addr { host: "127.10.250.65" port: 34845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:08.270088 31412 mvcc.cc:205] Tried to move back new op lower bound from 7298994127936155648 to 7298994127883190272. Current Snapshot: MvccSnapshot[applied={T|T < 7298994127936155648 or (T in {7298994127936155648})}]
I20260620 18:43:08.271775 30934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:08.271864 30934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:08.272692 30934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67ddeff7a33c46bab26eb422f7c51968: Sending DeleteTablet for 3 replicas of tablet 9465d5c178f24156a4da347bda108403
I20260620 18:43:08.273062 31056 tablet_service.cc:1514] Processing DeleteTablet for tablet 9465d5c178f24156a4da347bda108403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:08 UTC) from {username='slave'} at 127.0.0.1:40836
I20260620 18:43:08.273059 31187 tablet_service.cc:1514] Processing DeleteTablet for tablet 9465d5c178f24156a4da347bda108403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20260620 18:43:08.273288 31421 tablet_replica.cc:330] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260: stopping tablet replica
I20260620 18:43:08.273279 31318 tablet_service.cc:1514] Processing DeleteTablet for tablet 9465d5c178f24156a4da347bda108403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:08 UTC) from {username='slave'} at 127.0.0.1:39234
I20260620 18:43:08.273373 31421 raft_consensus.cc:2229] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:08.273517 31423 tablet_replica.cc:330] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c: stopping tablet replica
I20260620 18:43:08.273518 31421 raft_consensus.cc:2258] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:08.273602 31423 raft_consensus.cc:2229] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:43:08.273720 31423 raft_consensus.cc:2258] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:08.273913 31422 tablet_replica.cc:330] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640: stopping tablet replica
I20260620 18:43:08.273983 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 30994
I20260620 18:43:08.274050 31421 ts_tablet_manager.cc:1905] T 9465d5c178f24156a4da347bda108403 P 5a4922f55ec8493ea49bd598f5124260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:08.274145 31422 raft_consensus.cc:2229] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:08.274314 31422 raft_consensus.cc:2258] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:08.275046 31422 ts_tablet_manager.cc:1905] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:08.277030 31422 ts_tablet_manager.cc:1918] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:08.277166 31422 log.cc:1196] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-1/wal/wals/9465d5c178f24156a4da347bda108403
I20260620 18:43:08.277521 31422 ts_tablet_manager.cc:1939] T 9465d5c178f24156a4da347bda108403 P 0cc3a8c8f94344569fdb80206ada6640: Deleting consensus metadata
I20260620 18:43:08.278051 30920 catalog_manager.cc:4841] TS 0cc3a8c8f94344569fdb80206ada6640 (127.10.250.66:41903): tablet 9465d5c178f24156a4da347bda108403 (table test-table [id=c50d3926c3a6438eb9f44216902ce9f8]) successfully deleted
W20260620 18:43:08.280481 30919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.250.65:34845 (error 108)
I20260620 18:43:08.280756 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 31125
I20260620 18:43:08.281360 31392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:08.280870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.65:34845 (local address 127.0.0.1:40838)
0620 18:43:08.281275 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:08.281323 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.65:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384}
I20260620 18:43:08.281744 31423 ts_tablet_manager.cc:1905] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:08.283232 31423 ts_tablet_manager.cc:1918] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:08.283308 31423 log.cc:1196] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c: Deleting WAL directory at /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980738687848-11241-0/minicluster-data/ts-2/wal/wals/9465d5c178f24156a4da347bda108403
I20260620 18:43:08.283587 31423 ts_tablet_manager.cc:1939] T 9465d5c178f24156a4da347bda108403 P 33ff019bd2a14188a6de510fc848790c: Deleting consensus metadata
W20260620 18:43:08.283769 30919 catalog_manager.cc:4584] TS 5a4922f55ec8493ea49bd598f5124260 (127.10.250.65:34845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9465d5c178f24156a4da347bda108403: Network error: Client connection negotiation failed: client connection to 127.10.250.65:34845: connect: Connection refused (error 111)
I20260620 18:43:08.284844 30921 catalog_manager.cc:4841] TS 33ff019bd2a14188a6de510fc848790c (127.10.250.67:41447): tablet 9465d5c178f24156a4da347bda108403 (table test-table [id=c50d3926c3a6438eb9f44216902ce9f8]) successfully deleted
I20260620 18:43:08.286979 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 31256
I20260620 18:43:08.291857 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 30903
2026-06-20T18:43:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1024 ms)
[----------] 6 tests from EncryptionPolicyTest (9011 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:08 dist-test-slave-wp6g krb5kdc[31427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:08 dist-test-slave-wp6g krb5kdc[31427](info): set up 2 sockets
Jun 20 18:43:08 dist-test-slave-wp6g krb5kdc[31427](info): commencing operation
krb5kdc: starting...
W20260620 18:43:09.346593 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:09 dist-test-slave-wp6g krb5kdc[31427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:09.470707 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:09.556874 31440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:09.557174 31440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:09.557224 31440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:09.558770 31440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:09.558859 31440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:09.558885 31440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:09.558916 31440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:09.558946 31440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:09.560693 31440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36413
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:09.561026 31440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:09.561290 31440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:09.567023 31448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:09.567046 31446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:09.567049 31445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:09.567077 31440 server_base.cc:1034] running on GCE node
I20260620 18:43:09.567576 31440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:09.567889 31440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:09.569099 31440 hybrid_clock.cc:648] HybridClock initialized: now 1781980989569087 us; error 31 us; skew 500 ppm
Jun 20 18:43:09 dist-test-slave-wp6g krb5kdc[31427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:09.571904 31440 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:09.572415 31440 webserver.cc:460] Webserver started at http://127.0.0.1:38951/ using document root <none> and password file <none>
I20260620 18:43:09.572681 31440 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:09.572753 31440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:09.572906 31440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:09.573995 31440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "b76ca023c1014efc902fae4d7e97cb31"
format_stamp: "Formatted at 2026-06-20 18:43:09 on dist-test-slave-wp6g"
server_key: "84a65dd84a61fdd2f2e254947f0e57e1"
server_key_iv: "14ebe55d65644fc84cab6e335aa1d601"
server_key_version: "encryptionkey@0"
I20260620 18:43:09.574376 31440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "b76ca023c1014efc902fae4d7e97cb31"
format_stamp: "Formatted at 2026-06-20 18:43:09 on dist-test-slave-wp6g"
server_key: "84a65dd84a61fdd2f2e254947f0e57e1"
server_key_iv: "14ebe55d65644fc84cab6e335aa1d601"
server_key_version: "encryptionkey@0"
I20260620 18:43:09.575835 31440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:09.576673 31455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:09.576862 31440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:09.576967 31440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "b76ca023c1014efc902fae4d7e97cb31"
format_stamp: "Formatted at 2026-06-20 18:43:09 on dist-test-slave-wp6g"
server_key: "84a65dd84a61fdd2f2e254947f0e57e1"
server_key_iv: "14ebe55d65644fc84cab6e335aa1d601"
server_key_version: "encryptionkey@0"
I20260620 18:43:09.577057 31440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:09.584882 31440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:09.587530 31440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:09.587714 31440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:09.592242 31440 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36413
I20260620 18:43:09.592278 31507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36413 every 8 connection(s)
I20260620 18:43:09.592669 31440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:09.593593 31508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:09.595963 31508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31: Bootstrap starting.
I20260620 18:43:09.596315 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 31440
I20260620 18:43:09.596431 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:09.596632 11241 external_mini_cluster.cc:1428] Setting key ae8c77f2604bd7f8d8c87ebe55247dcb
I20260620 18:43:09.596719 31508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:09.597074 31508 log.cc:824] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:09.597854 31508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31: No bootstrap required, opened a new log
I20260620 18:43:09.599819 31508 raft_consensus.cc:348] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76ca023c1014efc902fae4d7e97cb31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36413 } }
I20260620 18:43:09.599988 31508 raft_consensus.cc:374] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:09.600039 31508 raft_consensus.cc:729] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76ca023c1014efc902fae4d7e97cb31, State: Initialized, Role: FOLLOWER
I20260620 18:43:09.600201 31508 consensus_queue.cc:260] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76ca023c1014efc902fae4d7e97cb31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36413 } }
I20260620 18:43:09.600301 31508 raft_consensus.cc:388] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:09.600346 31508 raft_consensus.cc:482] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:09.600395 31508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:09.600955 31508 raft_consensus.cc:504] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76ca023c1014efc902fae4d7e97cb31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36413 } }
I20260620 18:43:09.601075 31508 leader_election.cc:302] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b76ca023c1014efc902fae4d7e97cb31; no voters:
I20260620 18:43:09.601280 31508 leader_election.cc:288] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:09.601301 31513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:09.601563 31508 sys_catalog.cc:564] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:09.601518 31513 raft_consensus.cc:686] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [term 1 LEADER]: Becoming Leader. State: Replica: b76ca023c1014efc902fae4d7e97cb31, State: Running, Role: LEADER
I20260620 18:43:09.602066 31513 consensus_queue.cc:237] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76ca023c1014efc902fae4d7e97cb31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36413 } }
I20260620 18:43:09.602777 31514 sys_catalog.cc:455] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b76ca023c1014efc902fae4d7e97cb31. Latest consensus state: current_term: 1 leader_uuid: "b76ca023c1014efc902fae4d7e97cb31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76ca023c1014efc902fae4d7e97cb31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36413 } } }
I20260620 18:43:09.602877 31514 sys_catalog.cc:458] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:09.603150 31515 sys_catalog.cc:455] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b76ca023c1014efc902fae4d7e97cb31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76ca023c1014efc902fae4d7e97cb31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36413 } } }
I20260620 18:43:09.603451 31515 sys_catalog.cc:458] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31 [sys.catalog]: This master's current role is: LEADER
W20260620 18:43:09.603564 31527 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:43:09.603616 31527 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 20 18:43:09 dist-test-slave-wp6g krb5kdc[31427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:43:09.603726 31528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:09.603905 31528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:09.605643 31528 catalog_manager.cc:1347] Generated new cluster ID: 51bb4a10b6c0487fb0943e1b074f18ad
I20260620 18:43:09.605716 31528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:09.613061 31512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.597233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45773 (local address 127.0.0.1:36413)
0620 18:43:09.597460 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:09.597469 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:09.597509 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:43:09.598011 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:09.598018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:09.598075 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:09.598236 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:09.598958 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.599844 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.600774 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.601197 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.601346 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.604279 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:09.604284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:09.604285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:09.604299 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:09.611027 (+ 6728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:09.611629 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:09.611634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:09.611635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:09.611767 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:09.612021 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:09.612022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:09.612023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:09.612414 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:09.612585 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:09.612688 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 18:43:09.612802 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":90,"threads_started":1}
I20260620 18:43:09.614112 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:09.623660 31512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.615777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45273 (local address 127.0.0.1:36413)
0620 18:43:09.615819 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:09.615821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:09.615831 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:43:09.615853 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:09.615853 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:09.615864 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:09.615941 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:09.616589 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.617407 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.618443 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.618873 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.618948 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.620458 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:09.620459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:09.620459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:09.620467 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:09.622212 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:09.622687 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:09.622689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:09.622689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:09.622755 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:09.622995 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:09.622996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:09.622996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:09.623146 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:09.623254 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:09.623373 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 18:43:09.623445 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:43:09.627516 31528 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:09.628499 31528 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:09.642549 31528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b76ca023c1014efc902fae4d7e97cb31: Generated new TSK 0
I20260620 18:43:09.642839 31528 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:09.650226 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 31440
W20260620 18:43:09.654733 31425 connection.cc:535] client connection to 127.0.0.1:36413 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36413): Connection reset by peer (error 104)
2026-06-20T18:43:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:09 dist-test-slave-wp6g krb5kdc[31539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:09 dist-test-slave-wp6g krb5kdc[31539](info): set up 2 sockets
Jun 20 18:43:09 dist-test-slave-wp6g krb5kdc[31539](info): commencing operation
krb5kdc: starting...
W20260620 18:43:10.702733 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:10 dist-test-slave-wp6g krb5kdc[31539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:10.825491 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:10.911821 31552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:10.912111 31552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:10.912158 31552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:10.913461 31552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:10.913543 31552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:10.913569 31552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:10.913609 31552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:10.913642 31552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:10.915381 31552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35005
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:10.915706 31552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:10.915964 31552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:10.921507 31557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:10.921518 31560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:10.921526 31558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:10.921830 31552 server_base.cc:1034] running on GCE node
I20260620 18:43:10.922097 31552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:10.922408 31552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:10.923599 31552 hybrid_clock.cc:648] HybridClock initialized: now 1781980990923579 us; error 54 us; skew 500 ppm
Jun 20 18:43:10 dist-test-slave-wp6g krb5kdc[31539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:10.926141 31552 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:10.926762 31552 webserver.cc:460] Webserver started at http://127.0.0.1:33995/ using document root <none> and password file <none>
I20260620 18:43:10.927021 31552 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:10.927094 31552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:10.927248 31552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:10.928377 31552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "0547f4f1c2674028a08ca95da1cf3cf3"
format_stamp: "Formatted at 2026-06-20 18:43:10 on dist-test-slave-wp6g"
server_key: "4729461bc4b8ebdf4c499eb65a758162"
server_key_iv: "e6e8b3f5a0c0ff74dcbed3b0c420ef78"
server_key_version: "encryptionkey@0"
I20260620 18:43:10.928768 31552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "0547f4f1c2674028a08ca95da1cf3cf3"
format_stamp: "Formatted at 2026-06-20 18:43:10 on dist-test-slave-wp6g"
server_key: "4729461bc4b8ebdf4c499eb65a758162"
server_key_iv: "e6e8b3f5a0c0ff74dcbed3b0c420ef78"
server_key_version: "encryptionkey@0"
I20260620 18:43:10.930210 31552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:43:10.931129 31567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:10.931372 31552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:10.931526 31552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "0547f4f1c2674028a08ca95da1cf3cf3"
format_stamp: "Formatted at 2026-06-20 18:43:10 on dist-test-slave-wp6g"
server_key: "4729461bc4b8ebdf4c499eb65a758162"
server_key_iv: "e6e8b3f5a0c0ff74dcbed3b0c420ef78"
server_key_version: "encryptionkey@0"
I20260620 18:43:10.931614 31552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:10.944283 31552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:10.946314 31552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:10.946491 31552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:10.950920 31552 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35005
I20260620 18:43:10.950963 31619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35005 every 8 connection(s)
I20260620 18:43:10.951356 31552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:10.952244 31620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:10.954702 31620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3: Bootstrap starting.
I20260620 18:43:10.955312 31620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:10.955618 31620 log.cc:824] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:10.956411 31620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3: No bootstrap required, opened a new log
I20260620 18:43:10.958120 31620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0547f4f1c2674028a08ca95da1cf3cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35005 } }
I20260620 18:43:10.958268 31620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:10.958292 31620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0547f4f1c2674028a08ca95da1cf3cf3, State: Initialized, Role: FOLLOWER
I20260620 18:43:10.958472 31620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0547f4f1c2674028a08ca95da1cf3cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35005 } }
I20260620 18:43:10.958591 31620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:10.958643 31620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:10.958696 31620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:10.959276 31620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0547f4f1c2674028a08ca95da1cf3cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35005 } }
I20260620 18:43:10.959390 31620 leader_election.cc:302] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0547f4f1c2674028a08ca95da1cf3cf3; no voters:
I20260620 18:43:10.959584 31620 leader_election.cc:288] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:10.959741 31624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:10.959846 31620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:10.959868 31624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 0547f4f1c2674028a08ca95da1cf3cf3, State: Running, Role: LEADER
I20260620 18:43:10.959952 31624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0547f4f1c2674028a08ca95da1cf3cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35005 } }
I20260620 18:43:10.960515 31625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0547f4f1c2674028a08ca95da1cf3cf3. Latest consensus state: current_term: 1 leader_uuid: "0547f4f1c2674028a08ca95da1cf3cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0547f4f1c2674028a08ca95da1cf3cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35005 } } }
I20260620 18:43:10.960613 31625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:10.960803 31623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0547f4f1c2674028a08ca95da1cf3cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0547f4f1c2674028a08ca95da1cf3cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35005 } } }
I20260620 18:43:10.960882 31623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:10.961006 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 31552
I20260620 18:43:10.961112 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:10.961336 11241 external_mini_cluster.cc:1428] Setting key 6d036c31ee92c1f56663b49c705fab48
W20260620 18:43:10.961791 31639 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:43:10.961853 31639 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:43:10.961916 31632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:10.962095 31632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:10.963593 31632 catalog_manager.cc:1347] Generated new cluster ID: db13512fa97044b3b24115a9e61b29da
I20260620 18:43:10.963644 31632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:10.968204 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:10.961840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50799 (local address 127.0.0.1:35005)
0620 18:43:10.962663 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:10.962672 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:10.962705 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:43:10.962823 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:10.962830 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:10.962884 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:10.963022 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:10.963338 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:10.964735 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:10.965880 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:10.966548 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:10.966642 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:10.967461 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:10.967466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:10.967466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:10.967689 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:10.967728 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:10.967743 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:10.967826 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:10.967937 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 18:43:10.967965 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":78,"threads_started":1}
I20260620 18:43:10.969187 31632 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:10.969329 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:10.969822 31632 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:10.977517 31632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0547f4f1c2674028a08ca95da1cf3cf3: Generated new TSK 0
I20260620 18:43:10.977756 31632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:10.978657 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:10.970956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39031 (local address 127.0.0.1:35005)
0620 18:43:10.970993 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:10.970995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:10.974788 (+ 3793us) server_negotiation.cc:407] Connection header received
0620 18:43:10.974911 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:10.974913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:10.974927 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:10.975005 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:10.975180 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:10.976051 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:10.977123 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:10.977429 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:10.977512 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:10.978267 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:10.978268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:10.978268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:10.978272 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:10.978286 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:10.978290 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:10.978337 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:10.978491 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 18:43:10.978504 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:43:10.980820 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 31552
W20260620 18:43:10.984978 31537 connection.cc:535] client connection to 127.0.0.1:35005 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35005): Connection reset by peer (error 104)
2026-06-20T18:43:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:11 dist-test-slave-wp6g krb5kdc[31651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:11 dist-test-slave-wp6g krb5kdc[31651](info): set up 2 sockets
Jun 20 18:43:11 dist-test-slave-wp6g krb5kdc[31651](info): commencing operation
krb5kdc: starting...
W20260620 18:43:12.026748 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:12 dist-test-slave-wp6g krb5kdc[31651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:12.154289 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:12.238418 31664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:12.238691 31664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:12.238719 31664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:12.240041 31664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:12.240099 31664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:12.240114 31664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:12.240128 31664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:12.240140 31664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:12.241823 31664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:12.242113 31664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:12.242388 31664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:12.247442 31669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:12.247447 31670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:12.247447 31672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:12.247534 31664 server_base.cc:1034] running on GCE node
I20260620 18:43:12.247903 31664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:12.248210 31664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:12.249362 31664 hybrid_clock.cc:648] HybridClock initialized: now 1781980992249339 us; error 38 us; skew 500 ppm
Jun 20 18:43:12 dist-test-slave-wp6g krb5kdc[31651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:12.252121 31664 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:12.252552 31664 webserver.cc:460] Webserver started at http://127.0.0.1:33583/ using document root <none> and password file <none>
I20260620 18:43:12.252766 31664 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:12.252853 31664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:12.252960 31664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:12.254215 31664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "97488b550ef3452297a72de8c3477350"
format_stamp: "Formatted at 2026-06-20 18:43:12 on dist-test-slave-wp6g"
server_key: "7af8de00f42ccdacb14be3dabea2565f"
server_key_iv: "96fd32687b420a9a3e73d64b334e69a8"
server_key_version: "encryptionkey@0"
I20260620 18:43:12.254678 31664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "97488b550ef3452297a72de8c3477350"
format_stamp: "Formatted at 2026-06-20 18:43:12 on dist-test-slave-wp6g"
server_key: "7af8de00f42ccdacb14be3dabea2565f"
server_key_iv: "96fd32687b420a9a3e73d64b334e69a8"
server_key_version: "encryptionkey@0"
I20260620 18:43:12.256034 31664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:12.257056 31679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:12.257265 31664 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:12.257355 31664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "97488b550ef3452297a72de8c3477350"
format_stamp: "Formatted at 2026-06-20 18:43:12 on dist-test-slave-wp6g"
server_key: "7af8de00f42ccdacb14be3dabea2565f"
server_key_iv: "96fd32687b420a9a3e73d64b334e69a8"
server_key_version: "encryptionkey@0"
I20260620 18:43:12.257421 31664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:12.260972 31664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:12.263268 31664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:12.263382 31664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:12.267241 31664 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45371
I20260620 18:43:12.267293 31731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45371 every 8 connection(s)
I20260620 18:43:12.267638 31664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:12.268656 31732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:12.269476 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 31664
I20260620 18:43:12.269707 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:12.269924 11241 external_mini_cluster.cc:1428] Setting key 50d2f42ade06e7869b61c9f094887c75
I20260620 18:43:12.271831 31732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350: Bootstrap starting.
I20260620 18:43:12.272374 31732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:12.272614 31732 log.cc:824] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:12.273315 31732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350: No bootstrap required, opened a new log
I20260620 18:43:12.274712 31735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.270490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43089 (local address 127.0.0.1:45371)
0620 18:43:12.272867 (+ 2377us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:12.272875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:12.272899 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:43:12.273312 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:12.273319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:12.273344 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:12.273519 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:12.274197 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:12.274200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:12.274200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:12.274204 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:12.274243 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:12.274253 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:12.274321 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:12.274417 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 18:43:12.274482 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2332,"thread_start_us":123,"threads_started":1}
I20260620 18:43:12.275775 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:12.275933 31732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97488b550ef3452297a72de8c3477350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } }
I20260620 18:43:12.276093 31732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:12.276149 31732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97488b550ef3452297a72de8c3477350, State: Initialized, Role: FOLLOWER
I20260620 18:43:12.276271 31732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97488b550ef3452297a72de8c3477350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } }
I20260620 18:43:12.276378 31732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:12.276422 31732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:12.276473 31732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:12.277011 31732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97488b550ef3452297a72de8c3477350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } }
I20260620 18:43:12.277136 31732 leader_election.cc:302] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97488b550ef3452297a72de8c3477350; no voters:
I20260620 18:43:12.277319 31732 leader_election.cc:288] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:12.277352 31741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:12.277499 31741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [term 1 LEADER]: Becoming Leader. State: Replica: 97488b550ef3452297a72de8c3477350, State: Running, Role: LEADER
I20260620 18:43:12.277618 31732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:12.277583 31741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97488b550ef3452297a72de8c3477350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } }
I20260620 18:43:12.278003 31741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97488b550ef3452297a72de8c3477350. Latest consensus state: current_term: 1 leader_uuid: "97488b550ef3452297a72de8c3477350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97488b550ef3452297a72de8c3477350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } } }
I20260620 18:43:12.278044 31743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97488b550ef3452297a72de8c3477350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97488b550ef3452297a72de8c3477350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } } }
I20260620 18:43:12.278149 31741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:12.278374 31743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:12.278757 31751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:12.279356 31751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:12.279592 31735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.277311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57607 (local address 127.0.0.1:45371)
0620 18:43:12.277352 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:12.277354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:12.278344 (+ 990us) server_negotiation.cc:407] Connection header received
0620 18:43:12.278446 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:12.278447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:12.278452 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:12.278482 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:12.279283 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:12.279284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:12.279284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:12.279287 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:12.279321 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:12.279326 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:12.279383 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:12.279520 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 18:43:12.279532 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:43:12.280970 31751 catalog_manager.cc:1347] Generated new cluster ID: 91acc10ae7674f1bbd5867a65bf7f8a6
I20260620 18:43:12.281023 31751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:12.285985 31751 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:12.286168 31751 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:12.297778 31751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97488b550ef3452297a72de8c3477350: Generated new TSK 0
I20260620 18:43:12.297994 31751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:12.304067 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 31664
2026-06-20T18:43:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:12 dist-test-slave-wp6g krb5kdc[31763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:12 dist-test-slave-wp6g krb5kdc[31763](info): set up 2 sockets
Jun 20 18:43:12 dist-test-slave-wp6g krb5kdc[31763](info): commencing operation
krb5kdc: starting...
W20260620 18:43:13.362804 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:13 dist-test-slave-wp6g krb5kdc[31763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:13.485534 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:13.568758 31776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:13.569005 31776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:13.569031 31776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:13.570770 31776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:13.570839 31776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:13.570856 31776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:13.570870 31776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:13.570883 31776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:13.572659 31776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:13.572966 31776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:13.573230 31776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:13.578944 31781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:13.578967 31784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:13.578984 31776 server_base.cc:1034] running on GCE node
W20260620 18:43:13.578966 31782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:13.579581 31776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:13.579923 31776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:13.581127 31776 hybrid_clock.cc:648] HybridClock initialized: now 1781980993581112 us; error 30 us; skew 500 ppm
Jun 20 18:43:13 dist-test-slave-wp6g krb5kdc[31763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:13.584034 31776 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:13.584517 31776 webserver.cc:460] Webserver started at http://127.0.0.1:44719/ using document root <none> and password file <none>
I20260620 18:43:13.584762 31776 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:13.584810 31776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:13.584950 31776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:13.586129 31776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "00099f58c73840c0b5f6cb7be4901f38"
format_stamp: "Formatted at 2026-06-20 18:43:13 on dist-test-slave-wp6g"
server_key: "70e5179048709169f3bfabb9325a588a"
server_key_iv: "e69a6e0c4ba3081358c79db35f8e20a7"
server_key_version: "encryptionkey@0"
I20260620 18:43:13.586503 31776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "00099f58c73840c0b5f6cb7be4901f38"
format_stamp: "Formatted at 2026-06-20 18:43:13 on dist-test-slave-wp6g"
server_key: "70e5179048709169f3bfabb9325a588a"
server_key_iv: "e69a6e0c4ba3081358c79db35f8e20a7"
server_key_version: "encryptionkey@0"
I20260620 18:43:13.588091 31776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:13.588943 31791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:13.589211 31776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:13.589313 31776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "00099f58c73840c0b5f6cb7be4901f38"
format_stamp: "Formatted at 2026-06-20 18:43:13 on dist-test-slave-wp6g"
server_key: "70e5179048709169f3bfabb9325a588a"
server_key_iv: "e69a6e0c4ba3081358c79db35f8e20a7"
server_key_version: "encryptionkey@0"
I20260620 18:43:13.589394 31776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:13.599634 31776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:13.602227 31776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:13.602387 31776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:13.606474 31776 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34781
I20260620 18:43:13.606582 31843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34781 every 8 connection(s)
I20260620 18:43:13.606978 31776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:13.608019 31844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:13.610376 31844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38: Bootstrap starting.
I20260620 18:43:13.610921 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 31776
I20260620 18:43:13.611024 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:13.611131 31844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:13.611277 11241 external_mini_cluster.cc:1428] Setting key 5acf3dba625abb43d9958193187072a0
I20260620 18:43:13.611510 31844 log.cc:824] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:13.612332 31844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38: No bootstrap required, opened a new log
I20260620 18:43:13.614715 31844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00099f58c73840c0b5f6cb7be4901f38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34781 } }
I20260620 18:43:13.614902 31844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:13.614938 31844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00099f58c73840c0b5f6cb7be4901f38, State: Initialized, Role: FOLLOWER
I20260620 18:43:13.615080 31844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00099f58c73840c0b5f6cb7be4901f38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34781 } }
I20260620 18:43:13.615157 31844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:13.615188 31844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:13.615226 31844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:13.615923 31844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00099f58c73840c0b5f6cb7be4901f38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34781 } }
I20260620 18:43:13.616026 31844 leader_election.cc:302] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00099f58c73840c0b5f6cb7be4901f38; no voters:
I20260620 18:43:13.616221 31844 leader_election.cc:288] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:13.616273 31849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:13.616429 31849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [term 1 LEADER]: Becoming Leader. State: Replica: 00099f58c73840c0b5f6cb7be4901f38, State: Running, Role: LEADER
I20260620 18:43:13.616504 31844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:13.616526 31849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00099f58c73840c0b5f6cb7be4901f38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34781 } }
I20260620 18:43:13.616921 31849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00099f58c73840c0b5f6cb7be4901f38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00099f58c73840c0b5f6cb7be4901f38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34781 } } }
I20260620 18:43:13.617005 31849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:13.617147 31849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00099f58c73840c0b5f6cb7be4901f38. Latest consensus state: current_term: 1 leader_uuid: "00099f58c73840c0b5f6cb7be4901f38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00099f58c73840c0b5f6cb7be4901f38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34781 } } }
I20260620 18:43:13.617213 31849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:13.617336 31856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:13.617852 31856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 18:43:13 dist-test-slave-wp6g krb5kdc[31763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:43:13.619751 31856 catalog_manager.cc:1347] Generated new cluster ID: bc8183c763fa4a70a798900ae0ed7dd2
I20260620 18:43:13.619805 31856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:13.623317 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.611846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40771 (local address 127.0.0.1:34781)
0620 18:43:13.612057 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:13.612068 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:13.612104 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:43:13.612623 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:13.612630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:13.612679 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:13.612799 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:13.613609 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.614580 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.615682 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.616226 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.616350 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.619471 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:13.619476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:13.619477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:13.619489 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:13.621667 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:13.622066 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:13.622069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:13.622070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:13.622164 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:13.622385 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:13.622386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:13.622386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:13.622706 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:13.622875 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:13.622989 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 18:43:13.623136 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":117,"threads_started":1}
I20260620 18:43:13.624353 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:13.629562 31856 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:13.630309 31856 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:13.634490 31856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00099f58c73840c0b5f6cb7be4901f38: Generated new TSK 0
I20260620 18:43:13.634882 31856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:13.636454 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.625856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36617 (local address 127.0.0.1:34781)
0620 18:43:13.625897 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:13.625899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:13.629096 (+ 3197us) server_negotiation.cc:407] Connection header received
0620 18:43:13.629138 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:13.629139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:13.629152 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:13.629225 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:13.629981 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.630655 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.631409 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.631668 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.631752 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.633086 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:13.633088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:13.633088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:13.633096 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:13.635213 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:13.635629 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:13.635630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:13.635630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:13.635702 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:13.635909 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:13.635911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:13.635911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:13.636030 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:13.636132 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:13.636263 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 18:43:13.636355 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:43:13.638868 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 31776
2026-06-20T18:43:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:13 dist-test-slave-wp6g krb5kdc[31875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:13 dist-test-slave-wp6g krb5kdc[31875](info): set up 2 sockets
Jun 20 18:43:13 dist-test-slave-wp6g krb5kdc[31875](info): commencing operation
krb5kdc: starting...
W20260620 18:43:14.690757 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:14 dist-test-slave-wp6g krb5kdc[31875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:14.815009 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38951
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:14.901989 31888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:14.902212 31888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:14.902240 31888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:14.903512 31888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:14.903558 31888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:14.903574 31888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:14.903589 31888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:14.903601 31888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:14.905447 31888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:14.905665 31888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:14.905910 31888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:14.911010 31896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:14.911051 31888 server_base.cc:1034] running on GCE node
W20260620 18:43:14.911072 31893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:14.911032 31894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:14.911538 31888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:14.911877 31888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:14.913069 31888 hybrid_clock.cc:648] HybridClock initialized: now 1781980994913054 us; error 45 us; skew 500 ppm
Jun 20 18:43:14 dist-test-slave-wp6g krb5kdc[31875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:14.920279 31888 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:14.920717 31888 webserver.cc:460] Webserver started at http://127.0.0.1:33823/ using document root <none> and password file <none>
I20260620 18:43:14.920981 31888 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:14.921029 31888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:14.921180 31888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:14.922310 31888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "783228c099764bc09326ba5a7284cac7"
format_stamp: "Formatted at 2026-06-20 18:43:14 on dist-test-slave-wp6g"
server_key: "41f6d51bd784f949a259b983840541cc"
server_key_iv: "55e2b2de11228c37f65020f4f00124f5"
server_key_version: "encryptionkey@0"
I20260620 18:43:14.922730 31888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "783228c099764bc09326ba5a7284cac7"
format_stamp: "Formatted at 2026-06-20 18:43:14 on dist-test-slave-wp6g"
server_key: "41f6d51bd784f949a259b983840541cc"
server_key_iv: "55e2b2de11228c37f65020f4f00124f5"
server_key_version: "encryptionkey@0"
I20260620 18:43:14.924201 31888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:14.925159 31903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:14.925479 31888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:14.925626 31888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "783228c099764bc09326ba5a7284cac7"
format_stamp: "Formatted at 2026-06-20 18:43:14 on dist-test-slave-wp6g"
server_key: "41f6d51bd784f949a259b983840541cc"
server_key_iv: "55e2b2de11228c37f65020f4f00124f5"
server_key_version: "encryptionkey@0"
I20260620 18:43:14.925721 31888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:14.957618 31888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:14.960567 31888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:14.960741 31888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:14.964983 31888 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38951
I20260620 18:43:14.965029 31955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38951 every 8 connection(s)
I20260620 18:43:14.965432 31888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:14.966629 31956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:14.969381 31956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7: Bootstrap starting.
I20260620 18:43:14.969995 31956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:14.970288 31956 log.cc:824] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:14.970678 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 31888
I20260620 18:43:14.970808 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:14.971015 11241 external_mini_cluster.cc:1428] Setting key 6bdcff31fdaed363887393a9ae2f6be6
I20260620 18:43:14.971302 31956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7: No bootstrap required, opened a new log
I20260620 18:43:14.974884 31956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783228c099764bc09326ba5a7284cac7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38951 } }
I20260620 18:43:14.975097 31956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:14.975131 31956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783228c099764bc09326ba5a7284cac7, State: Initialized, Role: FOLLOWER
I20260620 18:43:14.975266 31956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783228c099764bc09326ba5a7284cac7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38951 } }
I20260620 18:43:14.975366 31956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:14.975428 31956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:14.975489 31956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:14.976212 31956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783228c099764bc09326ba5a7284cac7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38951 } }
I20260620 18:43:14.976328 31956 leader_election.cc:302] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 783228c099764bc09326ba5a7284cac7; no voters:
I20260620 18:43:14.976511 31956 leader_election.cc:288] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:14.976588 31961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:14.976778 31956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:14.976804 31961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [term 1 LEADER]: Becoming Leader. State: Replica: 783228c099764bc09326ba5a7284cac7, State: Running, Role: LEADER
I20260620 18:43:14.976900 31961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783228c099764bc09326ba5a7284cac7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38951 } }
I20260620 18:43:14.977483 31962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "783228c099764bc09326ba5a7284cac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783228c099764bc09326ba5a7284cac7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38951 } } }
I20260620 18:43:14.977595 31963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 783228c099764bc09326ba5a7284cac7. Latest consensus state: current_term: 1 leader_uuid: "783228c099764bc09326ba5a7284cac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783228c099764bc09326ba5a7284cac7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38951 } } }
I20260620 18:43:14.977643 31962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:14.977913 31963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:14.978116 31971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:14.978746 31960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:14.971640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46957 (local address 127.0.0.1:38951)
0620 18:43:14.971867 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:14.971880 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:14.971919 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:43:14.972970 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:14.972982 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:14.973057 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:14.973240 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:14.973715 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:14.975213 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:14.976141 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:14.976794 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:14.976911 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:14.977747 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:14.977751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:14.977751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:14.977970 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:14.978049 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:14.978068 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:14.978132 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:14.978453 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 18:43:14.978558 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
I20260620 18:43:14.979048 31971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:14.979697 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:14.980753 31971 catalog_manager.cc:1347] Generated new cluster ID: 825021de4770467c86b882cf52de5dbe
I20260620 18:43:14.980850 31971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:14.986357 31960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:14.981317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55175 (local address 127.0.0.1:38951)
0620 18:43:14.981351 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:14.981353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:14.982694 (+ 1341us) server_negotiation.cc:407] Connection header received
0620 18:43:14.982795 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:14.982796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:14.982807 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:14.982870 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:14.983046 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:14.983784 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:14.984849 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:14.985160 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:14.985221 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:14.986082 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:14.986084 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:14.986085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:14.986091 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:14.986108 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:14.986112 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:14.986162 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:14.986243 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 18:43:14.986271 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 18:43:14.990383 31971 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:14.991233 31971 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:14.996436 31971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 783228c099764bc09326ba5a7284cac7: Generated new TSK 0
I20260620 18:43:14.996694 31971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:14.999089 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 31888
2026-06-20T18:43:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:15 dist-test-slave-wp6g krb5kdc[31987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:15 dist-test-slave-wp6g krb5kdc[31987](info): set up 2 sockets
Jun 20 18:43:15 dist-test-slave-wp6g krb5kdc[31987](info): commencing operation
krb5kdc: starting...
W20260620 18:43:16.042675 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:16 dist-test-slave-wp6g krb5kdc[31987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:16.170607 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43119
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:16.258661 32000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:16.258889 32000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:16.258916 32000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:16.260125 32000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:16.260172 32000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:16.260187 32000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:16.260200 32000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:16.260213 32000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:16.261714 32000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:16.261921 32000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:16.262099 32000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:16.267495 32006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:16.267680 32000 server_base.cc:1034] running on GCE node
W20260620 18:43:16.267500 32008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:16.267500 32005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:16.268196 32000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:16.268558 32000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:16.269770 32000 hybrid_clock.cc:648] HybridClock initialized: now 1781980996269743 us; error 52 us; skew 500 ppm
Jun 20 18:43:16 dist-test-slave-wp6g krb5kdc[31987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:16.272572 32000 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:16.273068 32000 webserver.cc:460] Webserver started at http://127.0.0.1:38199/ using document root <none> and password file <none>
I20260620 18:43:16.273265 32000 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:16.273306 32000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:16.273401 32000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:16.274479 32000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "2dd164370ba44930a41cfa9ff66acb65"
format_stamp: "Formatted at 2026-06-20 18:43:16 on dist-test-slave-wp6g"
server_key: "650b57d2cda2b1aebf161302117816bf"
server_key_iv: "d73774b73c185792b866c3ab4dcc5d6e"
server_key_version: "encryptionkey@0"
I20260620 18:43:16.274868 32000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "2dd164370ba44930a41cfa9ff66acb65"
format_stamp: "Formatted at 2026-06-20 18:43:16 on dist-test-slave-wp6g"
server_key: "650b57d2cda2b1aebf161302117816bf"
server_key_iv: "d73774b73c185792b866c3ab4dcc5d6e"
server_key_version: "encryptionkey@0"
I20260620 18:43:16.276177 32000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:16.277007 32015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:16.277276 32000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:16.277405 32000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "2dd164370ba44930a41cfa9ff66acb65"
format_stamp: "Formatted at 2026-06-20 18:43:16 on dist-test-slave-wp6g"
server_key: "650b57d2cda2b1aebf161302117816bf"
server_key_iv: "d73774b73c185792b866c3ab4dcc5d6e"
server_key_version: "encryptionkey@0"
I20260620 18:43:16.277468 32000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:16.281117 32000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:16.283620 32000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:16.283738 32000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:16.288153 32000 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43119
I20260620 18:43:16.288187 32067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43119 every 8 connection(s)
I20260620 18:43:16.288545 32000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:16.289615 32068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:16.292029 32068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65: Bootstrap starting.
I20260620 18:43:16.292582 32068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:16.292812 32068 log.cc:824] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:16.293483 32068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65: No bootstrap required, opened a new log
I20260620 18:43:16.295094 32068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd164370ba44930a41cfa9ff66acb65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43119 } }
I20260620 18:43:16.295234 32068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:16.295257 32068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd164370ba44930a41cfa9ff66acb65, State: Initialized, Role: FOLLOWER
I20260620 18:43:16.295400 32068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd164370ba44930a41cfa9ff66acb65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43119 } }
I20260620 18:43:16.295481 32068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:16.295504 32068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:16.295545 32068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:16.296072 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 32000
I20260620 18:43:16.296129 32068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd164370ba44930a41cfa9ff66acb65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43119 } }
I20260620 18:43:16.296192 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:16.296231 32068 leader_election.cc:302] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dd164370ba44930a41cfa9ff66acb65; no voters:
I20260620 18:43:16.296389 11241 external_mini_cluster.cc:1428] Setting key 4f217df8e7889b84953c39283b523c95
I20260620 18:43:16.296406 32068 leader_election.cc:288] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:16.296495 32071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:16.296679 32068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:16.296713 32071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [term 1 LEADER]: Becoming Leader. State: Replica: 2dd164370ba44930a41cfa9ff66acb65, State: Running, Role: LEADER
I20260620 18:43:16.296965 32071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd164370ba44930a41cfa9ff66acb65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43119 } }
I20260620 18:43:16.297303 32072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dd164370ba44930a41cfa9ff66acb65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd164370ba44930a41cfa9ff66acb65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43119 } } }
I20260620 18:43:16.297375 32072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:16.297607 32073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dd164370ba44930a41cfa9ff66acb65. Latest consensus state: current_term: 1 leader_uuid: "2dd164370ba44930a41cfa9ff66acb65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd164370ba44930a41cfa9ff66acb65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43119 } } }
I20260620 18:43:16.297662 32073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:16.297999 32078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:16.298966 32078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:16.299492 32075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.296999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38821 (local address 127.0.0.1:43119)
0620 18:43:16.297172 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:16.297182 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:16.297327 (+ 145us) server_negotiation.cc:407] Connection header received
0620 18:43:16.297537 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:16.297544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:16.297581 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:16.297716 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:16.298903 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:16.298907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:16.298907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:16.298912 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:16.298959 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:16.298973 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:16.299039 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:16.299275 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 18:43:16.299317 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260620 18:43:16.300547 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:16.300699 32078 catalog_manager.cc:1347] Generated new cluster ID: 20db64dcc6864ea5bd57c7f7f88987cb
I20260620 18:43:16.300752 32078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:16.308311 32075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.306922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53803 (local address 127.0.0.1:43119)
0620 18:43:16.306989 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:16.306991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:16.307159 (+ 168us) server_negotiation.cc:407] Connection header received
0620 18:43:16.307241 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:16.307243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:16.307250 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:16.307307 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:16.308063 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:16.308065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:16.308065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:16.308069 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:16.308087 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:16.308091 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:16.308146 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:16.308200 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 18:43:16.308218 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:43:16.323630 32078 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:16.323870 32078 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:16.333422 32078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dd164370ba44930a41cfa9ff66acb65: Generated new TSK 0
I20260620 18:43:16.333659 32078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:16.344789 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 32000
2026-06-20T18:43:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:16 dist-test-slave-wp6g krb5kdc[32099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:16 dist-test-slave-wp6g krb5kdc[32099](info): set up 2 sockets
Jun 20 18:43:16 dist-test-slave-wp6g krb5kdc[32099](info): commencing operation
krb5kdc: starting...
W20260620 18:43:17.410832 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:17 dist-test-slave-wp6g krb5kdc[32099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:17.534561 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42279
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:17.618929 32112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:17.619164 32112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:17.619234 32112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:17.620474 32112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:17.620522 32112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:17.620537 32112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:17.620550 32112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:17.620563 32112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:17.622044 32112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:17.622251 32112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:17.622432 32112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:17.627895 32117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:17.628021 32118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:17.627895 32120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:17.628172 32112 server_base.cc:1034] running on GCE node
I20260620 18:43:17.628459 32112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:17.628815 32112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:17.630054 32112 hybrid_clock.cc:648] HybridClock initialized: now 1781980997630048 us; error 51 us; skew 500 ppm
Jun 20 18:43:17 dist-test-slave-wp6g krb5kdc[32099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:17.632961 32112 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:17.633499 32112 webserver.cc:460] Webserver started at http://127.0.0.1:46375/ using document root <none> and password file <none>
I20260620 18:43:17.633742 32112 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:17.633811 32112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:17.634009 32112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:17.635239 32112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "863044afcb9144c7956cb3b96a7a501b"
format_stamp: "Formatted at 2026-06-20 18:43:17 on dist-test-slave-wp6g"
server_key: "b685beca46c5e015805f5d1af2a57f43"
server_key_iv: "81ea393654380b9b25cd10b61e5b411b"
server_key_version: "encryptionkey@0"
I20260620 18:43:17.635627 32112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "863044afcb9144c7956cb3b96a7a501b"
format_stamp: "Formatted at 2026-06-20 18:43:17 on dist-test-slave-wp6g"
server_key: "b685beca46c5e015805f5d1af2a57f43"
server_key_iv: "81ea393654380b9b25cd10b61e5b411b"
server_key_version: "encryptionkey@0"
I20260620 18:43:17.637058 32112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:43:17.637832 32127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:17.638044 32112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:17.638154 32112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "863044afcb9144c7956cb3b96a7a501b"
format_stamp: "Formatted at 2026-06-20 18:43:17 on dist-test-slave-wp6g"
server_key: "b685beca46c5e015805f5d1af2a57f43"
server_key_iv: "81ea393654380b9b25cd10b61e5b411b"
server_key_version: "encryptionkey@0"
I20260620 18:43:17.638229 32112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:17.692303 32112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:17.694720 32112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:17.694908 32112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:17.698881 32112 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42279
I20260620 18:43:17.698938 32179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42279 every 8 connection(s)
I20260620 18:43:17.699370 32112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:17.700395 32180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:17.700919 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 32112
I20260620 18:43:17.701082 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:17.701318 11241 external_mini_cluster.cc:1428] Setting key 9caf94e06cefca3faa757730d88f5569
I20260620 18:43:17.703728 32180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b: Bootstrap starting.
I20260620 18:43:17.704422 32180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:17.704732 32180 log.cc:824] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:17.705440 32180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b: No bootstrap required, opened a new log
I20260620 18:43:17.707556 32180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863044afcb9144c7956cb3b96a7a501b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42279 } }
I20260620 18:43:17.707728 32180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:17.707783 32180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863044afcb9144c7956cb3b96a7a501b, State: Initialized, Role: FOLLOWER
I20260620 18:43:17.707933 32180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863044afcb9144c7956cb3b96a7a501b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42279 } }
I20260620 18:43:17.708010 32180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:17.708035 32180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:17.708106 32180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [term 0 FOLLOWER]: Advancing to term 1
Jun 20 18:43:17 dist-test-slave-wp6g krb5kdc[32099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:43:17.708771 32180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863044afcb9144c7956cb3b96a7a501b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42279 } }
I20260620 18:43:17.708897 32180 leader_election.cc:302] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 863044afcb9144c7956cb3b96a7a501b; no voters:
I20260620 18:43:17.709086 32180 leader_election.cc:288] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:17.709132 32185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:17.709327 32185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [term 1 LEADER]: Becoming Leader. State: Replica: 863044afcb9144c7956cb3b96a7a501b, State: Running, Role: LEADER
I20260620 18:43:17.709384 32180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:17.709421 32185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863044afcb9144c7956cb3b96a7a501b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42279 } }
I20260620 18:43:17.709750 32186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 863044afcb9144c7956cb3b96a7a501b. Latest consensus state: current_term: 1 leader_uuid: "863044afcb9144c7956cb3b96a7a501b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863044afcb9144c7956cb3b96a7a501b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42279 } } }
I20260620 18:43:17.709932 32187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "863044afcb9144c7956cb3b96a7a501b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863044afcb9144c7956cb3b96a7a501b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42279 } } }
I20260620 18:43:17.710224 32187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:17.710423 32186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:17.710826 32194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:17.711442 32194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:17.713014 32183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:17.701959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:49103 (local address 127.0.0.1:42279)
0620 18:43:17.702162 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:17.702173 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:17.702211 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:43:17.702757 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:17.702766 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:17.702828 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:17.702961 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:17.703874 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:17.704904 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:17.705730 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:17.706149 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:17.706272 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:17.708861 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:17.708866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:17.708866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:17.708880 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:17.711210 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:17.711721 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:17.711724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:17.711724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:17.711829 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:17.712101 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:17.712102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:17.712102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:17.712370 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:17.712554 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:17.712679 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 18:43:17.712783 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260620 18:43:17.713406 32194 catalog_manager.cc:1347] Generated new cluster ID: 3f541bff2fab423fbfab7fc2d5c3aa7d
I20260620 18:43:17.713457 32194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:17.713901 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:17.722569 32183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:17.715332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:37461 (local address 127.0.0.1:42279)
0620 18:43:17.715382 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:17.715383 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:17.715449 (+ 66us) server_negotiation.cc:407] Connection header received
0620 18:43:17.715530 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:17.715531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:17.715540 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:17.715565 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:17.716216 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:17.716939 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:17.717913 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:17.718250 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:17.718285 (+ 35us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:17.719787 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:17.719788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:17.719788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:17.719795 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:17.721350 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:17.721753 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:17.721753 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:17.721754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:17.721800 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:17.722006 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:17.722007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:17.722008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:17.722123 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:17.722190 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:17.722389 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 18:43:17.722450 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 18:43:17.734396 32194 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:17.735150 32194 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:17.751855 32194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 863044afcb9144c7956cb3b96a7a501b: Generated new TSK 0
I20260620 18:43:17.752110 32194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:17.757788 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 32112
W20260620 18:43:17.762015 32097 connection.cc:535] client connection to 127.0.0.1:42279 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42279): Connection reset by peer (error 104)
2026-06-20T18:43:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:17 dist-test-slave-wp6g krb5kdc[32211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:17 dist-test-slave-wp6g krb5kdc[32211](info): set up 2 sockets
Jun 20 18:43:17 dist-test-slave-wp6g krb5kdc[32211](info): commencing operation
krb5kdc: starting...
W20260620 18:43:18.818845 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:18 dist-test-slave-wp6g krb5kdc[32211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:18.945106 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:19.031736 32224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:19.031978 32224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:19.032006 32224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:19.033305 32224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:19.033355 32224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:19.033370 32224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:19.033383 32224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:19.033396 32224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:19.034983 32224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:19.035259 32224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:19.035527 32224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:43:19.040756 32224 server_base.cc:1034] running on GCE node
W20260620 18:43:19.040683 32229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:19.040683 32232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:19.040759 32230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:19.041256 32224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:19.041611 32224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:19.042807 32224 hybrid_clock.cc:648] HybridClock initialized: now 1781980999042785 us; error 33 us; skew 500 ppm
Jun 20 18:43:19 dist-test-slave-wp6g krb5kdc[32211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:19.045488 32224 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:19.045943 32224 webserver.cc:460] Webserver started at http://127.0.0.1:36675/ using document root <none> and password file <none>
I20260620 18:43:19.046216 32224 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:19.046289 32224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:19.046445 32224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:19.047570 32224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "e3479d968692435f9cb7e09678b1f884"
format_stamp: "Formatted at 2026-06-20 18:43:19 on dist-test-slave-wp6g"
server_key: "264f56bd90e738dc561f537e13578dea"
server_key_iv: "eff7895ad671007a5240645db143e819"
server_key_version: "encryptionkey@0"
I20260620 18:43:19.047941 32224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "e3479d968692435f9cb7e09678b1f884"
format_stamp: "Formatted at 2026-06-20 18:43:19 on dist-test-slave-wp6g"
server_key: "264f56bd90e738dc561f537e13578dea"
server_key_iv: "eff7895ad671007a5240645db143e819"
server_key_version: "encryptionkey@0"
I20260620 18:43:19.049413 32224 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:43:19.050256 32239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:19.050446 32224 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:19.050582 32224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "e3479d968692435f9cb7e09678b1f884"
format_stamp: "Formatted at 2026-06-20 18:43:19 on dist-test-slave-wp6g"
server_key: "264f56bd90e738dc561f537e13578dea"
server_key_iv: "eff7895ad671007a5240645db143e819"
server_key_version: "encryptionkey@0"
I20260620 18:43:19.050663 32224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:19.061175 32224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:19.063588 32224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:19.063753 32224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:19.067878 32224 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41561
I20260620 18:43:19.067951 32291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41561 every 8 connection(s)
I20260620 18:43:19.068289 32224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:19.069235 32292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:19.070199 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 32224
I20260620 18:43:19.070289 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:19.070592 11241 external_mini_cluster.cc:1428] Setting key 0c657c97bacd12f67c357954397da7c0
I20260620 18:43:19.072135 32292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884: Bootstrap starting.
I20260620 18:43:19.072788 32292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:19.073089 32292 log.cc:824] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:19.073818 32292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884: No bootstrap required, opened a new log
I20260620 18:43:19.075845 32292 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3479d968692435f9cb7e09678b1f884" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41561 } }
I20260620 18:43:19.076009 32292 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:19.076078 32292 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3479d968692435f9cb7e09678b1f884, State: Initialized, Role: FOLLOWER
I20260620 18:43:19.076242 32292 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3479d968692435f9cb7e09678b1f884" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41561 } }
I20260620 18:43:19.076340 32292 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:19.076385 32292 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:19.076428 32292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:19.076762 32295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:19.071180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:47299 (local address 127.0.0.1:41561)
0620 18:43:19.071364 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:19.071374 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:19.071401 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:43:19.072000 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:19.072008 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:19.072065 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:19.072280 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:19.072506 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:19.073559 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:19.074439 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:19.074955 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:19.075099 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:19.075935 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:19.075939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:19.075941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:19.076168 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:19.076232 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:19.076250 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:19.076335 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:19.076566 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 18:43:19.076601 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260620 18:43:19.077170 32292 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3479d968692435f9cb7e09678b1f884" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41561 } }
I20260620 18:43:19.077283 32292 leader_election.cc:302] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3479d968692435f9cb7e09678b1f884; no voters:
I20260620 18:43:19.077535 32292 leader_election.cc:288] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:19.077574 32297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:19.077749 32297 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [term 1 LEADER]: Becoming Leader. State: Replica: e3479d968692435f9cb7e09678b1f884, State: Running, Role: LEADER
I20260620 18:43:19.077811 32292 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:19.077845 32297 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3479d968692435f9cb7e09678b1f884" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41561 } }
I20260620 18:43:19.077906 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:19.078361 32298 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3479d968692435f9cb7e09678b1f884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3479d968692435f9cb7e09678b1f884" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41561 } } }
I20260620 18:43:19.078490 32298 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:19.078655 32298 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3479d968692435f9cb7e09678b1f884. Latest consensus state: current_term: 1 leader_uuid: "e3479d968692435f9cb7e09678b1f884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3479d968692435f9cb7e09678b1f884" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41561 } } }
I20260620 18:43:19.078713 32298 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:19.079041 32306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:19.079671 32306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:19.081364 32306 catalog_manager.cc:1347] Generated new cluster ID: 94cde0dca81a47f8b567ef99efcd62b1
I20260620 18:43:19.081431 32306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:19.083122 32295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:19.079614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:37667 (local address 127.0.0.1:41561)
0620 18:43:19.079649 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:19.079652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:19.079896 (+ 244us) server_negotiation.cc:407] Connection header received
0620 18:43:19.079976 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:19.079977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:19.079989 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:19.080046 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:19.080179 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:19.080871 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:19.081807 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:19.082030 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:19.082108 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:19.082888 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:19.082889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:19.082890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:19.082895 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:19.082910 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:19.082913 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:19.082967 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:19.083031 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 18:43:19.083041 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 18:43:19.103179 32306 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:19.103933 32306 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:19.115741 32306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3479d968692435f9cb7e09678b1f884: Generated new TSK 0
I20260620 18:43:19.116078 32306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:19.122618 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 32224
W20260620 18:43:19.127030 32209 connection.cc:535] client connection to 127.0.0.1:41561 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41561): Connection reset by peer (error 104)
2026-06-20T18:43:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:19 dist-test-slave-wp6g krb5kdc[32323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:19 dist-test-slave-wp6g krb5kdc[32323](info): set up 2 sockets
Jun 20 18:43:19 dist-test-slave-wp6g krb5kdc[32323](info): commencing operation
krb5kdc: starting...
W20260620 18:43:20.174925 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:20 dist-test-slave-wp6g krb5kdc[32323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:20.306118 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:20.390630 32336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:20.390914 32336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:20.390960 32336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:20.392355 32336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:20.392443 32336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:20.392469 32336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:20.392490 32336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:20.392520 32336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:20.394121 32336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:20.394474 32336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:20.394748 32336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:20.399967 32341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:20.399984 32344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:20.399952 32342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:20.400296 32336 server_base.cc:1034] running on GCE node
I20260620 18:43:20.400621 32336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:20.400961 32336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:20.402149 32336 hybrid_clock.cc:648] HybridClock initialized: now 1781981000402130 us; error 33 us; skew 500 ppm
Jun 20 18:43:20 dist-test-slave-wp6g krb5kdc[32323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:20.409475 32336 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:20.410035 32336 webserver.cc:460] Webserver started at http://127.0.0.1:33239/ using document root <none> and password file <none>
I20260620 18:43:20.410251 32336 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:20.410295 32336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:20.410406 32336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:20.411538 32336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "de974326745942708af13267bf30e9fc"
format_stamp: "Formatted at 2026-06-20 18:43:20 on dist-test-slave-wp6g"
server_key: "d846140e1ec5154ac1cf935b261f2481"
server_key_iv: "f696ae80596f1cfc7d2de81291da8ec0"
server_key_version: "encryptionkey@0"
I20260620 18:43:20.411870 32336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "de974326745942708af13267bf30e9fc"
format_stamp: "Formatted at 2026-06-20 18:43:20 on dist-test-slave-wp6g"
server_key: "d846140e1ec5154ac1cf935b261f2481"
server_key_iv: "f696ae80596f1cfc7d2de81291da8ec0"
server_key_version: "encryptionkey@0"
I20260620 18:43:20.413278 32336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:43:20.414142 32351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:20.414403 32336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:20.414494 32336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "de974326745942708af13267bf30e9fc"
format_stamp: "Formatted at 2026-06-20 18:43:20 on dist-test-slave-wp6g"
server_key: "d846140e1ec5154ac1cf935b261f2481"
server_key_iv: "f696ae80596f1cfc7d2de81291da8ec0"
server_key_version: "encryptionkey@0"
I20260620 18:43:20.414634 32336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:20.418004 32336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:20.420105 32336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:20.420222 32336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:20.424032 32336 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45383
I20260620 18:43:20.424104 32403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45383 every 8 connection(s)
I20260620 18:43:20.424459 32336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:20.425576 32404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:20.428097 32404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc: Bootstrap starting.
I20260620 18:43:20.428671 32404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:20.428922 32404 log.cc:824] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:20.429713 32404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc: No bootstrap required, opened a new log
I20260620 18:43:20.431527 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 32336
I20260620 18:43:20.431689 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:20.431550 32404 raft_consensus.cc:348] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de974326745942708af13267bf30e9fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } }
I20260620 18:43:20.431736 32404 raft_consensus.cc:374] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:20.431815 32404 raft_consensus.cc:729] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de974326745942708af13267bf30e9fc, State: Initialized, Role: FOLLOWER
I20260620 18:43:20.431918 11241 external_mini_cluster.cc:1428] Setting key f26c3e2434ef3f60ebe5b9710c350eab
I20260620 18:43:20.432025 32404 consensus_queue.cc:260] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de974326745942708af13267bf30e9fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } }
I20260620 18:43:20.432150 32404 raft_consensus.cc:388] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:20.432191 32404 raft_consensus.cc:482] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:20.432246 32404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:20.433475 32404 raft_consensus.cc:504] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de974326745942708af13267bf30e9fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } }
I20260620 18:43:20.433616 32404 leader_election.cc:302] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de974326745942708af13267bf30e9fc; no voters:
I20260620 18:43:20.433810 32404 leader_election.cc:288] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:20.433862 32409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:20.434010 32409 raft_consensus.cc:686] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [term 1 LEADER]: Becoming Leader. State: Replica: de974326745942708af13267bf30e9fc, State: Running, Role: LEADER
I20260620 18:43:20.434095 32404 sys_catalog.cc:564] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:20.434137 32409 consensus_queue.cc:237] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de974326745942708af13267bf30e9fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } }
I20260620 18:43:20.434556 32408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:20.432532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:50559 (local address 127.0.0.1:45383)
0620 18:43:20.432716 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:20.432726 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:20.432764 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:43:20.432892 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:20.432899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:20.432922 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:20.433048 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:20.433820 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:20.433824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:20.433824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:20.433830 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:20.433893 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:20.433912 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:20.433980 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:20.434225 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 18:43:20.434305 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260620 18:43:20.434661 32410 sys_catalog.cc:455] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader de974326745942708af13267bf30e9fc. Latest consensus state: current_term: 1 leader_uuid: "de974326745942708af13267bf30e9fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de974326745942708af13267bf30e9fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } } }
I20260620 18:43:20.434880 32410 sys_catalog.cc:458] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:20.434661 32411 sys_catalog.cc:455] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de974326745942708af13267bf30e9fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de974326745942708af13267bf30e9fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } } }
I20260620 18:43:20.435043 32411 sys_catalog.cc:458] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:20.436041 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 18:43:20.436399 32424 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:43:20.437249 32424 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:43:20.437414 32418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:20.437638 32418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:20.438611 32408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:20.437338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:60937 (local address 127.0.0.1:45383)
0620 18:43:20.437378 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:20.437380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:20.437508 (+ 128us) server_negotiation.cc:407] Connection header received
0620 18:43:20.437592 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:20.437593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:20.437599 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:20.437672 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:20.438345 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:20.438347 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:20.438347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:20.438351 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:20.438367 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:20.438372 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:20.438434 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:20.438499 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 18:43:20.438545 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:43:20.443142 32418 catalog_manager.cc:1347] Generated new cluster ID: 73c40abe9d9143ef8228aa39dfdf9fe3
I20260620 18:43:20.443213 32418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:20.479602 32418 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:20.479826 32418 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:20.484071 32418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de974326745942708af13267bf30e9fc: Generated new TSK 0
I20260620 18:43:20.484318 32418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:20.489773 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 32336
2026-06-20T18:43:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:20 dist-test-slave-wp6g krb5kdc[32435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:20 dist-test-slave-wp6g krb5kdc[32435](info): set up 2 sockets
Jun 20 18:43:20 dist-test-slave-wp6g krb5kdc[32435](info): commencing operation
krb5kdc: starting...
W20260620 18:43:21.539039 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:21 dist-test-slave-wp6g krb5kdc[32435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:21.662058 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:21.743830 32448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:21.744103 32448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:21.744145 32448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:21.745391 32448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:21.745471 32448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:21.745496 32448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:21.745517 32448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:21.745545 32448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:21.747071 32448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41705
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:21.747355 32448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:21.747586 32448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:21.752612 32456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:21.752645 32448 server_base.cc:1034] running on GCE node
W20260620 18:43:21.752624 32454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:21.752601 32453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:21.753170 32448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:21.753435 32448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:21.754593 32448 hybrid_clock.cc:648] HybridClock initialized: now 1781981001754575 us; error 31 us; skew 500 ppm
Jun 20 18:43:21 dist-test-slave-wp6g krb5kdc[32435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:21.757334 32448 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:21.757854 32448 webserver.cc:460] Webserver started at http://127.0.0.1:43411/ using document root <none> and password file <none>
I20260620 18:43:21.758085 32448 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:21.758154 32448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:21.758311 32448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:21.759436 32448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "87c29b9c26bd4f8281a3910baf106b1e"
format_stamp: "Formatted at 2026-06-20 18:43:21 on dist-test-slave-wp6g"
server_key: "a74e4722d2207293af532cba25949a80"
server_key_iv: "5f21416258c39f76160736529c8634f1"
server_key_version: "encryptionkey@0"
I20260620 18:43:21.759814 32448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "87c29b9c26bd4f8281a3910baf106b1e"
format_stamp: "Formatted at 2026-06-20 18:43:21 on dist-test-slave-wp6g"
server_key: "a74e4722d2207293af532cba25949a80"
server_key_iv: "5f21416258c39f76160736529c8634f1"
server_key_version: "encryptionkey@0"
I20260620 18:43:21.761127 32448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:43:21.761885 32463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:21.762073 32448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:43:21.762178 32448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "87c29b9c26bd4f8281a3910baf106b1e"
format_stamp: "Formatted at 2026-06-20 18:43:21 on dist-test-slave-wp6g"
server_key: "a74e4722d2207293af532cba25949a80"
server_key_iv: "5f21416258c39f76160736529c8634f1"
server_key_version: "encryptionkey@0"
I20260620 18:43:21.762251 32448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:21.779187 32448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:21.781661 32448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:21.781847 32448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:21.786276 32448 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41705
I20260620 18:43:21.786339 32515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41705 every 8 connection(s)
I20260620 18:43:21.786765 32448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:21.787367 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 32448
I20260620 18:43:21.787511 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:21.787758 11241 external_mini_cluster.cc:1428] Setting key 8d646d08f80a58b9857906900fbeb0aa
I20260620 18:43:21.788030 32516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:21.791916 32516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e: Bootstrap starting.
I20260620 18:43:21.792670 32516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:21.793025 32516 log.cc:824] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:21.793916 32516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e: No bootstrap required, opened a new log
Jun 20 18:43:21 dist-test-slave-wp6g krb5kdc[32435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:43:21.796754 32516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c29b9c26bd4f8281a3910baf106b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } }
I20260620 18:43:21.796976 32516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:21.797029 32516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c29b9c26bd4f8281a3910baf106b1e, State: Initialized, Role: FOLLOWER
I20260620 18:43:21.797186 32516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c29b9c26bd4f8281a3910baf106b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } }
I20260620 18:43:21.797279 32516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:21.797305 32516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:21.797438 32516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:21.798170 32516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c29b9c26bd4f8281a3910baf106b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } }
I20260620 18:43:21.798269 32516 leader_election.cc:302] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87c29b9c26bd4f8281a3910baf106b1e; no voters:
I20260620 18:43:21.798565 32516 leader_election.cc:288] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:21.798605 32521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:21.798755 32521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [term 1 LEADER]: Becoming Leader. State: Replica: 87c29b9c26bd4f8281a3910baf106b1e, State: Running, Role: LEADER
I20260620 18:43:21.798882 32516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:21.798915 32521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c29b9c26bd4f8281a3910baf106b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } }
I20260620 18:43:21.799301 32521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87c29b9c26bd4f8281a3910baf106b1e. Latest consensus state: current_term: 1 leader_uuid: "87c29b9c26bd4f8281a3910baf106b1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c29b9c26bd4f8281a3910baf106b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } } }
I20260620 18:43:21.799315 32522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87c29b9c26bd4f8281a3910baf106b1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c29b9c26bd4f8281a3910baf106b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } } }
I20260620 18:43:21.799482 32521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:21.799496 32522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:21.799731 32529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:21.800313 32529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:21.800845 32519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:21.788367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:33079 (local address 127.0.0.1:41705)
0620 18:43:21.788586 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:21.788598 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:21.788637 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:43:21.789409 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:21.789419 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:21.789496 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:21.789672 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:21.790303 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:21.791854 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:21.792658 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:21.793357 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:21.793451 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:21.796570 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:21.796574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:21.796574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:21.796585 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:21.798595 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:21.799130 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:21.799133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:21.799133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:21.799245 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:21.799525 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:21.799526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:21.799526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:21.799857 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:21.800466 (+ 609us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:21.800553 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 18:43:21.800652 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":99,"threads_started":1}
I20260620 18:43:21.801904 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:21.803186 32529 catalog_manager.cc:1347] Generated new cluster ID: e9b2f28ace9e4d46a8c9185ac6ec15e5
I20260620 18:43:21.803243 32529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:21.811535 32519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:21.803287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:34299 (local address 127.0.0.1:41705)
0620 18:43:21.803339 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:21.803341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:21.803497 (+ 156us) server_negotiation.cc:407] Connection header received
0620 18:43:21.803563 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:21.803564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:21.803575 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:21.803624 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:21.804347 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:21.805064 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:21.806227 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:21.806583 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:21.806640 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:21.808291 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:21.808293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:21.808293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:21.808301 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:21.810074 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:21.810624 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:21.810625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:21.810625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:21.810686 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:21.810967 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:21.810968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:21.810969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:21.811095 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:21.811186 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:21.811373 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 18:43:21.811439 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 18:43:21.813741 32529 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:21.814373 32529 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:21.818650 32529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87c29b9c26bd4f8281a3910baf106b1e: Generated new TSK 0
I20260620 18:43:21.818853 32529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:21.821979 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 32448
W20260620 18:43:21.826320 32433 connection.cc:535] client connection to 127.0.0.1:41705 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41705): Connection reset by peer (error 104)
2026-06-20T18:43:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:21 dist-test-slave-wp6g krb5kdc[32547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:21 dist-test-slave-wp6g krb5kdc[32547](info): set up 2 sockets
Jun 20 18:43:21 dist-test-slave-wp6g krb5kdc[32547](info): commencing operation
krb5kdc: starting...
W20260620 18:43:22.878871 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:22 dist-test-slave-wp6g krb5kdc[32547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:23.006892 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:23.093043 32560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:23.093302 32560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:23.093346 32560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:23.094614 32560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:23.094689 32560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:23.094717 32560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:23.094750 32560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:23.094774 32560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:23.096340 32560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45393
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:23.096668 32560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:23.096917 32560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:23.102041 32566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:23.102043 32568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:23.102124 32560 server_base.cc:1034] running on GCE node
W20260620 18:43:23.102054 32565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:23.102622 32560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:23.102957 32560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:23.104111 32560 hybrid_clock.cc:648] HybridClock initialized: now 1781981003104093 us; error 40 us; skew 500 ppm
Jun 20 18:43:23 dist-test-slave-wp6g krb5kdc[32547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:23.106731 32560 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:23.107203 32560 webserver.cc:460] Webserver started at http://127.0.0.1:35079/ using document root <none> and password file <none>
I20260620 18:43:23.107456 32560 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:23.107528 32560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:23.107678 32560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:23.108804 32560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "4325514ea19e422ea22755b7cca49d90"
format_stamp: "Formatted at 2026-06-20 18:43:23 on dist-test-slave-wp6g"
server_key: "26d8a347fb8fbe4a148c43dbca81ac81"
server_key_iv: "08a4242a04f74bee8960fb4b94b90e0a"
server_key_version: "encryptionkey@0"
I20260620 18:43:23.109175 32560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "4325514ea19e422ea22755b7cca49d90"
format_stamp: "Formatted at 2026-06-20 18:43:23 on dist-test-slave-wp6g"
server_key: "26d8a347fb8fbe4a148c43dbca81ac81"
server_key_iv: "08a4242a04f74bee8960fb4b94b90e0a"
server_key_version: "encryptionkey@0"
I20260620 18:43:23.110637 32560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:43:23.111465 32575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:23.111634 32560 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:43:23.111729 32560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "4325514ea19e422ea22755b7cca49d90"
format_stamp: "Formatted at 2026-06-20 18:43:23 on dist-test-slave-wp6g"
server_key: "26d8a347fb8fbe4a148c43dbca81ac81"
server_key_iv: "08a4242a04f74bee8960fb4b94b90e0a"
server_key_version: "encryptionkey@0"
I20260620 18:43:23.111817 32560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:23.133258 32560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:23.135879 32560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:23.136058 32560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:23.140307 32560 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45393
I20260620 18:43:23.140388 32627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45393 every 8 connection(s)
I20260620 18:43:23.140866 32560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:23.142119 32628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:23.142724 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 32560
I20260620 18:43:23.142868 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:23.143102 11241 external_mini_cluster.cc:1428] Setting key 0cf2896dd1a594603ea669f1e0ab86ab
I20260620 18:43:23.148680 32628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90: Bootstrap starting.
I20260620 18:43:23.149742 32628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:23.150137 32628 log.cc:824] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:23.151253 32628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90: No bootstrap required, opened a new log
I20260620 18:43:23.151463 32631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:23.143732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:54271 (local address 127.0.0.1:45393)
0620 18:43:23.143962 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:23.143974 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:23.144013 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:43:23.145122 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:23.145133 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:23.145212 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:23.145677 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:23.145784 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:23.147373 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:23.148773 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:23.149486 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:23.150222 (+ 736us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:23.150598 (+ 376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:23.150602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:23.150602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:23.150819 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:23.150869 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:23.150886 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:23.151120 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:23.151166 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 18:43:23.151197 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":96,"threads_started":1}
I20260620 18:43:23.154441 32628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4325514ea19e422ea22755b7cca49d90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } }
I20260620 18:43:23.154671 32628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:23.154706 32628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325514ea19e422ea22755b7cca49d90, State: Initialized, Role: FOLLOWER
I20260620 18:43:23.154708 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:23.154855 32628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4325514ea19e422ea22755b7cca49d90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } }
I20260620 18:43:23.154922 32628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:23.154950 32628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:23.154989 32628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:23.156003 32628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4325514ea19e422ea22755b7cca49d90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } }
I20260620 18:43:23.156118 32628 leader_election.cc:302] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4325514ea19e422ea22755b7cca49d90; no voters:
I20260620 18:43:23.156270 32628 leader_election.cc:288] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:23.156486 32639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:23.156520 32628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:23.156594 32639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [term 1 LEADER]: Becoming Leader. State: Replica: 4325514ea19e422ea22755b7cca49d90, State: Running, Role: LEADER
I20260620 18:43:23.156818 32639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4325514ea19e422ea22755b7cca49d90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } }
I20260620 18:43:23.157146 32639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4325514ea19e422ea22755b7cca49d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4325514ea19e422ea22755b7cca49d90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } } }
I20260620 18:43:23.157238 32639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:23.157383 32639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4325514ea19e422ea22755b7cca49d90. Latest consensus state: current_term: 1 leader_uuid: "4325514ea19e422ea22755b7cca49d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4325514ea19e422ea22755b7cca49d90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } } }
I20260620 18:43:23.157437 32639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:23.160859 32631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:23.157005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:49069 (local address 127.0.0.1:45393)
0620 18:43:23.157043 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:23.157045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:23.157052 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:43:23.157074 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:23.157075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:23.157084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:23.157300 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:23.157328 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:23.158032 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:23.159061 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:23.159351 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:23.159694 (+ 343us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:23.160458 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:23.160460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:23.160460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:23.160465 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:23.160481 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:23.160485 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:23.160736 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:23.160761 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 18:43:23.160772 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:43:23.161361 32645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:23.161815 32645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:23.163455 32645 catalog_manager.cc:1347] Generated new cluster ID: f6711982f40a409f858ea5e893aa29b2
I20260620 18:43:23.163503 32645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:23.191710 32645 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:23.192523 32645 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:23.196223 32645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4325514ea19e422ea22755b7cca49d90: Generated new TSK 0
I20260620 18:43:23.196445 32645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:23.203142 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 32560
W20260620 18:43:23.207834 32545 connection.cc:535] client connection to 127.0.0.1:45393 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45393): Connection reset by peer (error 104)
2026-06-20T18:43:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:23 dist-test-slave-wp6g krb5kdc[32659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:23 dist-test-slave-wp6g krb5kdc[32659](info): set up 2 sockets
Jun 20 18:43:23 dist-test-slave-wp6g krb5kdc[32659](info): commencing operation
krb5kdc: starting...
W20260620 18:43:24.250830 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:24 dist-test-slave-wp6g krb5kdc[32659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:24.377763 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33147
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:24.461014 32672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:24.461288 32672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:24.461330 32672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:24.462646 32672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:24.462718 32672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:24.462742 32672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:24.462772 32672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:24.462803 32672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:24.464458 32672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33147
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:24.464783 32672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:24.465041 32672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:24.470160 32678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:24.470270 32672 server_base.cc:1034] running on GCE node
W20260620 18:43:24.470187 32680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:24.470172 32677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:24.470685 32672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:24.471050 32672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:24.472254 32672 hybrid_clock.cc:648] HybridClock initialized: now 1781981004472234 us; error 37 us; skew 500 ppm
Jun 20 18:43:24 dist-test-slave-wp6g krb5kdc[32659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:24.475375 32672 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:24.475926 32672 webserver.cc:460] Webserver started at http://127.0.0.1:37667/ using document root <none> and password file <none>
I20260620 18:43:24.476172 32672 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:24.476248 32672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:24.476408 32672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:24.477528 32672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "5f30504c16f34d37affc37400c2d69a2"
format_stamp: "Formatted at 2026-06-20 18:43:24 on dist-test-slave-wp6g"
server_key: "eb8df7b5f56690ca43886840b16c857d"
server_key_iv: "5df24cca64c95992cb77f7b5db109891"
server_key_version: "encryptionkey@0"
I20260620 18:43:24.477927 32672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "5f30504c16f34d37affc37400c2d69a2"
format_stamp: "Formatted at 2026-06-20 18:43:24 on dist-test-slave-wp6g"
server_key: "eb8df7b5f56690ca43886840b16c857d"
server_key_iv: "5df24cca64c95992cb77f7b5db109891"
server_key_version: "encryptionkey@0"
I20260620 18:43:24.479440 32672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:43:24.480269 32687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:24.480568 32672 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:24.480693 32672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "5f30504c16f34d37affc37400c2d69a2"
format_stamp: "Formatted at 2026-06-20 18:43:24 on dist-test-slave-wp6g"
server_key: "eb8df7b5f56690ca43886840b16c857d"
server_key_iv: "5df24cca64c95992cb77f7b5db109891"
server_key_version: "encryptionkey@0"
I20260620 18:43:24.480783 32672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:24.484510 32672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:24.486954 32672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:24.487124 32672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:24.491786 32672 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33147
I20260620 18:43:24.491819 32739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33147 every 8 connection(s)
I20260620 18:43:24.492182 32672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:24.493089 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 32672
I20260620 18:43:24.493268 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:24.493187 32740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:24.493507 11241 external_mini_cluster.cc:1428] Setting key c1a7dd9fdf4cbae069a2426a9b46af57
I20260620 18:43:24.496244 32743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:24.494126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:52147 (local address 127.0.0.1:33147)
0620 18:43:24.494350 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:24.494358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:24.494385 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:43:24.494912 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:24.494919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:24.494949 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:24.495075 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:24.495784 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:24.495787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:24.495788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:24.495792 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:24.495830 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:24.495841 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:24.495896 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:24.495985 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 18:43:24.496044 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
I20260620 18:43:24.496884 32740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2: Bootstrap starting.
I20260620 18:43:24.497228 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:24.497619 32740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:24.497912 32740 log.cc:824] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:24.498744 32740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2: No bootstrap required, opened a new log
I20260620 18:43:24.499568 32743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:24.498562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:45661 (local address 127.0.0.1:33147)
0620 18:43:24.498601 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:24.498603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:24.498773 (+ 170us) server_negotiation.cc:407] Connection header received
0620 18:43:24.498814 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:24.498814 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:24.498818 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:24.498864 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:24.499360 (+ 496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:24.499361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:24.499361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:24.499364 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:24.499375 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:24.499378 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:24.499414 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:24.499473 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 18:43:24.499484 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:43:24.501009 32740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f30504c16f34d37affc37400c2d69a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } }
I20260620 18:43:24.501129 32740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:24.501150 32740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f30504c16f34d37affc37400c2d69a2, State: Initialized, Role: FOLLOWER
I20260620 18:43:24.501243 32740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f30504c16f34d37affc37400c2d69a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } }
I20260620 18:43:24.501296 32740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:24.501317 32740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:24.501343 32740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:24.501864 32740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f30504c16f34d37affc37400c2d69a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } }
I20260620 18:43:24.501941 32740 leader_election.cc:302] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f30504c16f34d37affc37400c2d69a2; no voters:
I20260620 18:43:24.502064 32740 leader_election.cc:288] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:24.502139 32750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:24.502295 32740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:24.502315 32750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [term 1 LEADER]: Becoming Leader. State: Replica: 5f30504c16f34d37affc37400c2d69a2, State: Running, Role: LEADER
I20260620 18:43:24.502401 32750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f30504c16f34d37affc37400c2d69a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } }
I20260620 18:43:24.502817 32752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f30504c16f34d37affc37400c2d69a2. Latest consensus state: current_term: 1 leader_uuid: "5f30504c16f34d37affc37400c2d69a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f30504c16f34d37affc37400c2d69a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } } }
I20260620 18:43:24.502813 32751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f30504c16f34d37affc37400c2d69a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f30504c16f34d37affc37400c2d69a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } } }
I20260620 18:43:24.502931 32752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:24.502936 32751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:24.503165 32759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:24.503671 32759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:24.505501 32759 catalog_manager.cc:1347] Generated new cluster ID: 3711b851d02646f78c1bdd547c9a139e
I20260620 18:43:24.505563 32759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:24.513312 32759 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:24.513587 32759 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:24.519214 32759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f30504c16f34d37affc37400c2d69a2: Generated new TSK 0
I20260620 18:43:24.519472 32759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:24.528358 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 32672
2026-06-20T18:43:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:24 dist-test-slave-wp6g krb5kdc[303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:24 dist-test-slave-wp6g krb5kdc[303](info): set up 2 sockets
Jun 20 18:43:24 dist-test-slave-wp6g krb5kdc[303](info): commencing operation
krb5kdc: starting...
W20260620 18:43:25.570932 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:25 dist-test-slave-wp6g krb5kdc[303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:43:25.701207 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44525
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:45349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:44525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:25.787370 316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:25.787612 316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:25.787653 316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:25.788837 316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:25.788908 316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:25.788932 316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:25.788959 316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:25.788991 316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:25.790578 316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:45349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:44525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:25.790867 316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:25.791102 316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:43:25.796202 316 server_base.cc:1034] running on GCE node
W20260620 18:43:25.796155 324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:25.796173 321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:25.796314 322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:25.796590 316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:25.796957 316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:25.798130 316 hybrid_clock.cc:648] HybridClock initialized: now 1781981005798126 us; error 42 us; skew 500 ppm
Jun 20 18:43:25 dist-test-slave-wp6g krb5kdc[303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981005, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:25.805613 316 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:43:25.806238 316 webserver.cc:460] Webserver started at http://127.10.250.126:46413/ using document root <none> and password file <none>
I20260620 18:43:25.806500 316 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:25.806602 316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:25.806744 316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:25.807901 316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "ebb1034b551f4bfdbedb6648ac2b1e33"
format_stamp: "Formatted at 2026-06-20 18:43:25 on dist-test-slave-wp6g"
server_key: "183e1b61aca474dd85a259a632770828"
server_key_iv: "c2c124d5c9070dfd3b142fb96bb6ce04"
server_key_version: "encryptionkey@0"
I20260620 18:43:25.808291 316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "ebb1034b551f4bfdbedb6648ac2b1e33"
format_stamp: "Formatted at 2026-06-20 18:43:25 on dist-test-slave-wp6g"
server_key: "183e1b61aca474dd85a259a632770828"
server_key_iv: "c2c124d5c9070dfd3b142fb96bb6ce04"
server_key_version: "encryptionkey@0"
I20260620 18:43:25.809770 316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:43:25.810700 331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:25.810917 316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:25.811126 316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "ebb1034b551f4bfdbedb6648ac2b1e33"
format_stamp: "Formatted at 2026-06-20 18:43:25 on dist-test-slave-wp6g"
server_key: "183e1b61aca474dd85a259a632770828"
server_key_iv: "c2c124d5c9070dfd3b142fb96bb6ce04"
server_key_version: "encryptionkey@0"
I20260620 18:43:25.811218 316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:25.819149 316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:25.821352 316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:25.821547 316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:25.825838 316 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:44525
I20260620 18:43:25.825873 383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:44525 every 8 connection(s)
I20260620 18:43:25.826248 316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:25.826416 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 316
I20260620 18:43:25.826581 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:25.826861 11241 external_mini_cluster.cc:1428] Setting key 3214314b868e5ef7af88738c185d2202
I20260620 18:43:25.827402 384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:25.830741 384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33: Bootstrap starting.
I20260620 18:43:25.831413 384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:25.831710 384 log.cc:824] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:25.832655 384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33: No bootstrap required, opened a new log
Jun 20 18:43:25 dist-test-slave-wp6g krb5kdc[303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:25.835088 384 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebb1034b551f4bfdbedb6648ac2b1e33" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44525 } }
I20260620 18:43:25.835287 384 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:25.835317 384 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb1034b551f4bfdbedb6648ac2b1e33, State: Initialized, Role: FOLLOWER
I20260620 18:43:25.835482 384 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebb1034b551f4bfdbedb6648ac2b1e33" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44525 } }
I20260620 18:43:25.835551 384 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:25.835572 384 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:25.835641 384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:25.836220 384 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebb1034b551f4bfdbedb6648ac2b1e33" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44525 } }
I20260620 18:43:25.836308 384 leader_election.cc:302] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebb1034b551f4bfdbedb6648ac2b1e33; no voters:
I20260620 18:43:25.836508 384 leader_election.cc:288] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:25.836540 389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:25.836644 389 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [term 1 LEADER]: Becoming Leader. State: Replica: ebb1034b551f4bfdbedb6648ac2b1e33, State: Running, Role: LEADER
I20260620 18:43:25.836750 389 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebb1034b551f4bfdbedb6648ac2b1e33" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44525 } }
I20260620 18:43:25.836793 384 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:25.837217 389 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebb1034b551f4bfdbedb6648ac2b1e33. Latest consensus state: current_term: 1 leader_uuid: "ebb1034b551f4bfdbedb6648ac2b1e33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebb1034b551f4bfdbedb6648ac2b1e33" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44525 } } }
I20260620 18:43:25.837301 389 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:25.837533 390 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebb1034b551f4bfdbedb6648ac2b1e33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebb1034b551f4bfdbedb6648ac2b1e33" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44525 } } }
I20260620 18:43:25.837611 390 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:25.837517 397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:25.838397 397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:25.839248 387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:25.827395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37453 (local address 127.10.250.126:44525)
0620 18:43:25.827594 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:25.827603 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:25.827635 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:43:25.828202 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:25.828212 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:25.828283 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:25.828453 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:25.829103 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:25.830483 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:25.831405 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:25.832027 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:25.832112 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:25.834917 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:25.834922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:25.834922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:25.834935 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:25.837261 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:25.837794 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:25.837797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:25.837797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:25.837963 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:25.838286 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:25.838288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:25.838289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:25.838659 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:25.838837 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:25.838923 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 18:43:25.839046 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
I20260620 18:43:25.840342 397 catalog_manager.cc:1347] Generated new cluster ID: 8957f1bdfbed46ad99d6f8912974e974
I20260620 18:43:25.840395 397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:25.840520 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:25.854462 387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:25.842889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51961 (local address 127.10.250.126:44525)
0620 18:43:25.842929 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:25.842932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:25.846731 (+ 3799us) server_negotiation.cc:407] Connection header received
0620 18:43:25.846774 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:25.846775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:25.846787 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:25.846858 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:25.847524 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:25.848343 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:25.849347 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:25.849713 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:25.849789 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:25.851315 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:25.851316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:25.851316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:25.851324 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:25.853011 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:25.853462 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:25.853463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:25.853464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:25.853528 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:25.853748 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:25.853748 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:25.853749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:25.853887 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:25.853994 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:25.854121 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 18:43:25.854349 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:43:25.864460 397 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:25.865058 397 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:25.868016 397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebb1034b551f4bfdbedb6648ac2b1e33: Generated new TSK 0
I20260620 18:43:25.868274 397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:25.876999 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 316
W20260620 18:43:25.881471 301 connection.cc:535] client connection to 127.10.250.126:44525 recv error: Network error: failed to read from TLS socket (remote: 127.10.250.126:44525): Connection reset by peer (error 104)
2026-06-20T18:43:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:25 dist-test-slave-wp6g krb5kdc[415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:25 dist-test-slave-wp6g krb5kdc[415](info): set up 2 sockets
Jun 20 18:43:25 dist-test-slave-wp6g krb5kdc[415](info): commencing operation
krb5kdc: starting...
W20260620 18:43:26.914947 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:27 dist-test-slave-wp6g krb5kdc[415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:43:27.042611 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:34221
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:34221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:27.135301 428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:27.135533 428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:27.135560 428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:27.136821 428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:27.136876 428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:27.136893 428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:27.136905 428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:27.136945 428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:27.138690 428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:34221
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:34221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:27.138976 428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:27.139242 428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:27.145054 433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:27.145076 436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:27.145054 434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:27.145567 428 server_base.cc:1034] running on GCE node
I20260620 18:43:27.146030 428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:27.146351 428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:27.147563 428 hybrid_clock.cc:648] HybridClock initialized: now 1781981007147573 us; error 58 us; skew 500 ppm
Jun 20 18:43:27 dist-test-slave-wp6g krb5kdc[415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981007, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:27.150362 428 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:43:27.150884 428 webserver.cc:460] Webserver started at http://127.10.250.126:38259/ using document root <none> and password file <none>
I20260620 18:43:27.151152 428 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:27.151226 428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:27.151391 428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:27.152519 428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "af7b7c4c9d4f4310929ea47bfdf3488d"
format_stamp: "Formatted at 2026-06-20 18:43:27 on dist-test-slave-wp6g"
server_key: "dac2e8463a58a50fd085be39a3fd6cb8"
server_key_iv: "66a66e38b746996cc7adb641108b2e14"
server_key_version: "encryptionkey@0"
I20260620 18:43:27.152904 428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "af7b7c4c9d4f4310929ea47bfdf3488d"
format_stamp: "Formatted at 2026-06-20 18:43:27 on dist-test-slave-wp6g"
server_key: "dac2e8463a58a50fd085be39a3fd6cb8"
server_key_iv: "66a66e38b746996cc7adb641108b2e14"
server_key_version: "encryptionkey@0"
I20260620 18:43:27.154390 428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:43:27.155241 443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:27.155474 428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:27.155594 428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "af7b7c4c9d4f4310929ea47bfdf3488d"
format_stamp: "Formatted at 2026-06-20 18:43:27 on dist-test-slave-wp6g"
server_key: "dac2e8463a58a50fd085be39a3fd6cb8"
server_key_iv: "66a66e38b746996cc7adb641108b2e14"
server_key_version: "encryptionkey@0"
I20260620 18:43:27.155689 428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:27.176522 428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:27.179483 428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:27.179697 428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:27.183996 428 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:34221
I20260620 18:43:27.184038 495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:34221 every 8 connection(s)
I20260620 18:43:27.184444 428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:27.185413 496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:27.187948 496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d: Bootstrap starting.
I20260620 18:43:27.188555 496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:27.188807 496 log.cc:824] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:27.189580 496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d: No bootstrap required, opened a new log
I20260620 18:43:27.191385 496 raft_consensus.cc:348] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af7b7c4c9d4f4310929ea47bfdf3488d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 34221 } }
I20260620 18:43:27.191568 496 raft_consensus.cc:374] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:27.191594 496 raft_consensus.cc:729] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7b7c4c9d4f4310929ea47bfdf3488d, State: Initialized, Role: FOLLOWER
I20260620 18:43:27.191783 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 428
I20260620 18:43:27.191784 496 consensus_queue.cc:260] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af7b7c4c9d4f4310929ea47bfdf3488d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 34221 } }
I20260620 18:43:27.191880 496 raft_consensus.cc:388] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:27.191921 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:27.191920 496 raft_consensus.cc:482] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:27.191992 496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:27.192126 11241 external_mini_cluster.cc:1428] Setting key f0e8c26c10728f25faaf941389d74692
I20260620 18:43:27.193451 496 raft_consensus.cc:504] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af7b7c4c9d4f4310929ea47bfdf3488d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 34221 } }
I20260620 18:43:27.193586 496 leader_election.cc:302] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af7b7c4c9d4f4310929ea47bfdf3488d; no voters:
I20260620 18:43:27.193785 496 leader_election.cc:288] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:27.193846 501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:27.194012 501 raft_consensus.cc:686] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [term 1 LEADER]: Becoming Leader. State: Replica: af7b7c4c9d4f4310929ea47bfdf3488d, State: Running, Role: LEADER
I20260620 18:43:27.194083 496 sys_catalog.cc:564] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:27.194109 501 consensus_queue.cc:237] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af7b7c4c9d4f4310929ea47bfdf3488d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 34221 } }
I20260620 18:43:27.194474 503 sys_catalog.cc:455] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [sys.catalog]: SysCatalogTable state changed. Reason: New leader af7b7c4c9d4f4310929ea47bfdf3488d. Latest consensus state: current_term: 1 leader_uuid: "af7b7c4c9d4f4310929ea47bfdf3488d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af7b7c4c9d4f4310929ea47bfdf3488d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 34221 } } }
I20260620 18:43:27.194594 503 sys_catalog.cc:458] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:27.194700 502 sys_catalog.cc:455] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af7b7c4c9d4f4310929ea47bfdf3488d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af7b7c4c9d4f4310929ea47bfdf3488d" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 34221 } } }
I20260620 18:43:27.194880 502 sys_catalog.cc:458] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d [sys.catalog]: This master's current role is: LEADER
W20260620 18:43:27.196085 516 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:43:27.196146 516 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:43:27.196192 508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:27.196353 508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:27.198274 508 catalog_manager.cc:1347] Generated new cluster ID: 3c7c24a6297e43a9a1c7eecd02cd0f39
I20260620 18:43:27.198333 508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:27.201757 500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:27.192898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52155 (local address 127.10.250.126:34221)
0620 18:43:27.193202 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:27.193213 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:27.193252 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:43:27.193407 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:27.193415 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:27.193481 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:27.193662 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:27.194221 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:27.196090 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:27.197060 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:27.197765 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:27.197887 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:27.201043 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:27.201048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:27.201048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:27.201248 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:27.201287 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:27.201300 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:27.201402 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:27.201495 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 18:43:27.201530 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":116,"threads_started":1}
I20260620 18:43:27.202836 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:27.209909 500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:27.204279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44837 (local address 127.10.250.126:34221)
0620 18:43:27.204319 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:27.204321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:27.206778 (+ 2457us) server_negotiation.cc:407] Connection header received
0620 18:43:27.206890 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:27.206891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:27.206903 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:27.206955 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:27.207129 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:27.207674 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:27.208463 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:27.208708 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:27.208791 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:27.209599 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:27.209601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:27.209601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:27.209607 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:27.209624 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:27.209629 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:27.209701 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:27.209765 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 18:43:27.209779 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:43:27.212977 508 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:27.213583 508 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:27.218665 508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af7b7c4c9d4f4310929ea47bfdf3488d: Generated new TSK 0
I20260620 18:43:27.218890 508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:27.225327 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 428
W20260620 18:43:27.230849 413 connection.cc:535] client connection to 127.10.250.126:34221 recv error: Network error: failed to read from TLS socket (remote: 127.10.250.126:34221): Connection reset by peer (error 104)
2026-06-20T18:43:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:27 dist-test-slave-wp6g krb5kdc[527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:27 dist-test-slave-wp6g krb5kdc[527](info): set up 2 sockets
Jun 20 18:43:27 dist-test-slave-wp6g krb5kdc[527](info): commencing operation
krb5kdc: starting...
W20260620 18:43:28.322809 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:28 dist-test-slave-wp6g krb5kdc[527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:43:28.445600 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:39923
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:38513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:39923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:28.528471 540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:28.528708 540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:28.528734 540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:28.529901 540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:28.529943 540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:28.529958 540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:28.529971 540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:28.529983 540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:28.531618 540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:38513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:39923
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:39923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:28.531905 540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:28.532173 540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:28.537425 548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:28.537416 545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:28.537472 540 server_base.cc:1034] running on GCE node
W20260620 18:43:28.537421 546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:28.537860 540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:28.538136 540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:28.539275 540 hybrid_clock.cc:648] HybridClock initialized: now 1781981008539261 us; error 28 us; skew 500 ppm
Jun 20 18:43:28 dist-test-slave-wp6g krb5kdc[527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981008, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:28.542027 540 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:43:28.542619 540 webserver.cc:460] Webserver started at http://127.10.250.126:44461/ using document root <none> and password file <none>
I20260620 18:43:28.542892 540 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:28.542949 540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:28.543090 540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:28.544271 540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "aa392decda1c483a8c571c2b40cdf191"
format_stamp: "Formatted at 2026-06-20 18:43:28 on dist-test-slave-wp6g"
server_key: "518ed5b7b13fb380a711091b20eb7391"
server_key_iv: "d68f24cd1f6c01198fd36324acaba81d"
server_key_version: "encryptionkey@0"
I20260620 18:43:28.544641 540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "aa392decda1c483a8c571c2b40cdf191"
format_stamp: "Formatted at 2026-06-20 18:43:28 on dist-test-slave-wp6g"
server_key: "518ed5b7b13fb380a711091b20eb7391"
server_key_iv: "d68f24cd1f6c01198fd36324acaba81d"
server_key_version: "encryptionkey@0"
I20260620 18:43:28.546208 540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:43:28.547196 555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:28.547451 540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:28.547567 540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "aa392decda1c483a8c571c2b40cdf191"
format_stamp: "Formatted at 2026-06-20 18:43:28 on dist-test-slave-wp6g"
server_key: "518ed5b7b13fb380a711091b20eb7391"
server_key_iv: "d68f24cd1f6c01198fd36324acaba81d"
server_key_version: "encryptionkey@0"
I20260620 18:43:28.547655 540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:28.551345 540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:28.553859 540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:28.554031 540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:28.558379 540 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:39923
I20260620 18:43:28.558411 607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:39923 every 8 connection(s)
I20260620 18:43:28.558853 540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:28.559887 608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:28.560592 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 540
I20260620 18:43:28.560716 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:28.560916 11241 external_mini_cluster.cc:1428] Setting key 7ba4ff9d9b1599aa8d3b23310ac159bb
I20260620 18:43:28.563215 608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191: Bootstrap starting.
I20260620 18:43:28.563524 611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:28.561507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55591 (local address 127.10.250.126:39923)
0620 18:43:28.561683 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:28.561693 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:28.561729 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:43:28.562140 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:28.562146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:28.562172 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:28.562305 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:28.563022 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:28.563025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:28.563025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:28.563029 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:28.563067 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:28.563078 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:28.563134 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:28.563224 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 18:43:28.563320 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260620 18:43:28.564013 608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:28.564364 608 log.cc:824] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:28.564709 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:28.565148 608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191: No bootstrap required, opened a new log
I20260620 18:43:28.567364 608 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa392decda1c483a8c571c2b40cdf191" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39923 } }
I20260620 18:43:28.567564 608 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:28.567610 608 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa392decda1c483a8c571c2b40cdf191, State: Initialized, Role: FOLLOWER
I20260620 18:43:28.567672 611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:28.566391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54517 (local address 127.10.250.126:39923)
0620 18:43:28.566455 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:28.566458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:28.566653 (+ 195us) server_negotiation.cc:407] Connection header received
0620 18:43:28.566718 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:28.566719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:28.566723 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:28.566773 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:28.567387 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:28.567388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:28.567388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:28.567392 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:28.567408 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:28.567411 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:28.567458 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:28.567527 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 18:43:28.567538 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:43:28.567755 608 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa392decda1c483a8c571c2b40cdf191" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39923 } }
I20260620 18:43:28.567850 608 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:28.567875 608 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:28.567904 608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:28.568449 608 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa392decda1c483a8c571c2b40cdf191" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39923 } }
I20260620 18:43:28.568522 608 leader_election.cc:302] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa392decda1c483a8c571c2b40cdf191; no voters:
I20260620 18:43:28.568650 608 leader_election.cc:288] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:28.568715 618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:28.568868 618 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [term 1 LEADER]: Becoming Leader. State: Replica: aa392decda1c483a8c571c2b40cdf191, State: Running, Role: LEADER
I20260620 18:43:28.568892 608 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:28.568953 618 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa392decda1c483a8c571c2b40cdf191" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39923 } }
I20260620 18:43:28.569342 619 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa392decda1c483a8c571c2b40cdf191" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa392decda1c483a8c571c2b40cdf191" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39923 } } }
I20260620 18:43:28.569443 619 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:28.569403 620 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa392decda1c483a8c571c2b40cdf191. Latest consensus state: current_term: 1 leader_uuid: "aa392decda1c483a8c571c2b40cdf191" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa392decda1c483a8c571c2b40cdf191" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 39923 } } }
I20260620 18:43:28.569592 620 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:28.570354 627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:28.570891 627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:28.572441 627 catalog_manager.cc:1347] Generated new cluster ID: 9c95771783894b709a2915e2e497b1fc
I20260620 18:43:28.572510 627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:28.582435 627 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:28.582676 627 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:28.585742 627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa392decda1c483a8c571c2b40cdf191: Generated new TSK 0
I20260620 18:43:28.585937 627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:28.588943 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 540
2026-06-20T18:43:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:28 dist-test-slave-wp6g krb5kdc[639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:28 dist-test-slave-wp6g krb5kdc[639](info): set up 2 sockets
Jun 20 18:43:28 dist-test-slave-wp6g krb5kdc[639](info): commencing operation
krb5kdc: starting...
W20260620 18:43:29.650853 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:29 dist-test-slave-wp6g krb5kdc[639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:43:29.777305 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:45395
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:33607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:45395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:29.863368 652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:29.863593 652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:29.863621 652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:29.864945 652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:29.865000 652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:29.865020 652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:29.865033 652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:29.865046 652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:29.866819 652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:33607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:45395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:45395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:29.867115 652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:29.867408 652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:43:29.873374 652 server_base.cc:1034] running on GCE node
W20260620 18:43:29.873461 657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:29.873446 660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:29.873451 658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:29.874173 652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:29.874500 652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:29.875739 652 hybrid_clock.cc:648] HybridClock initialized: now 1781981009875705 us; error 50 us; skew 500 ppm
Jun 20 18:43:29 dist-test-slave-wp6g krb5kdc[639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981009, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:29.878490 652 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:43:29.879036 652 webserver.cc:460] Webserver started at http://127.10.250.126:38295/ using document root <none> and password file <none>
I20260620 18:43:29.879292 652 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:29.879499 652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:29.879679 652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:29.880865 652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "960cd2695aa843deb9eca3c1f66ae657"
format_stamp: "Formatted at 2026-06-20 18:43:29 on dist-test-slave-wp6g"
server_key: "f0b0d027f4ed1ec809978acdb8ae77b6"
server_key_iv: "4fa444475e7e6af242e440209765c9c3"
server_key_version: "encryptionkey@0"
I20260620 18:43:29.881258 652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "960cd2695aa843deb9eca3c1f66ae657"
format_stamp: "Formatted at 2026-06-20 18:43:29 on dist-test-slave-wp6g"
server_key: "f0b0d027f4ed1ec809978acdb8ae77b6"
server_key_iv: "4fa444475e7e6af242e440209765c9c3"
server_key_version: "encryptionkey@0"
I20260620 18:43:29.883008 652 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260620 18:43:29.883903 667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:29.884084 652 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:29.884223 652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "960cd2695aa843deb9eca3c1f66ae657"
format_stamp: "Formatted at 2026-06-20 18:43:29 on dist-test-slave-wp6g"
server_key: "f0b0d027f4ed1ec809978acdb8ae77b6"
server_key_iv: "4fa444475e7e6af242e440209765c9c3"
server_key_version: "encryptionkey@0"
I20260620 18:43:29.884313 652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:29.893146 652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:29.895354 652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:29.895529 652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:29.899633 652 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:45395
I20260620 18:43:29.899680 719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:45395 every 8 connection(s)
I20260620 18:43:29.900049 652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:29.900911 720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:29.902666 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 652
I20260620 18:43:29.902807 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:29.902997 11241 external_mini_cluster.cc:1428] Setting key da9afa0ddec734e223bda0e792845d9c
I20260620 18:43:29.903463 720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657: Bootstrap starting.
I20260620 18:43:29.904245 720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:29.904567 720 log.cc:824] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:29.905400 720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657: No bootstrap required, opened a new log
I20260620 18:43:29.907428 720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960cd2695aa843deb9eca3c1f66ae657" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 45395 } }
I20260620 18:43:29.907600 720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:29.907647 720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 960cd2695aa843deb9eca3c1f66ae657, State: Initialized, Role: FOLLOWER
I20260620 18:43:29.907796 720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960cd2695aa843deb9eca3c1f66ae657" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 45395 } }
I20260620 18:43:29.907896 720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:29.907936 720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:29.907991 720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:29.908823 720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960cd2695aa843deb9eca3c1f66ae657" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 45395 } }
I20260620 18:43:29.908949 720 leader_election.cc:302] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 960cd2695aa843deb9eca3c1f66ae657; no voters:
I20260620 18:43:29.909133 720 leader_election.cc:288] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:29.909160 725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:29.909353 725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [term 1 LEADER]: Becoming Leader. State: Replica: 960cd2695aa843deb9eca3c1f66ae657, State: Running, Role: LEADER
I20260620 18:43:29.909427 720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:29.909479 725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960cd2695aa843deb9eca3c1f66ae657" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 45395 } }
I20260620 18:43:29.909862 726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "960cd2695aa843deb9eca3c1f66ae657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960cd2695aa843deb9eca3c1f66ae657" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 45395 } } }
I20260620 18:43:29.909853 725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 960cd2695aa843deb9eca3c1f66ae657. Latest consensus state: current_term: 1 leader_uuid: "960cd2695aa843deb9eca3c1f66ae657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960cd2695aa843deb9eca3c1f66ae657" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 45395 } } }
I20260620 18:43:29.909945 726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:29.909953 725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:29.910177 731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 18:43:29 dist-test-slave-wp6g krb5kdc[639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:29.910833 731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:29.912501 731 catalog_manager.cc:1347] Generated new cluster ID: 709634b6a42f483db754a0691be9613d
I20260620 18:43:29.912557 731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:29.914988 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:29.903607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41315 (local address 127.10.250.126:45395)
0620 18:43:29.903797 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:29.903807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:29.903845 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:43:29.904489 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:29.904501 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:29.904577 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:29.904739 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:29.905638 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:29.906640 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:29.907416 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:29.907921 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:29.908035 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:29.911157 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:29.911161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:29.911162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:29.911174 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:29.913284 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:29.913673 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:29.913676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:29.913676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:29.913773 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:29.913948 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:29.913949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:29.913949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:29.914223 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:29.914363 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:29.914646 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 18:43:29.914782 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260620 18:43:29.916224 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:29.923059 731 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:29.924114 731 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:29.924968 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:29.917634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48209 (local address 127.10.250.126:45395)
0620 18:43:29.917665 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:29.917667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:29.917791 (+ 124us) server_negotiation.cc:407] Connection header received
0620 18:43:29.917852 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:29.917852 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:29.917862 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:29.917917 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:29.918596 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:29.919332 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:29.920366 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:29.920692 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:29.920750 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:29.922220 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:29.922221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:29.922222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:29.922229 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:29.923765 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:29.924121 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:29.924122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:29.924122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:29.924204 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:29.924425 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:29.924426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:29.924426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:29.924538 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:29.924636 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:29.924766 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 18:43:29.924861 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":14}
I20260620 18:43:29.938716 731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 960cd2695aa843deb9eca3c1f66ae657: Generated new TSK 0
I20260620 18:43:29.938975 731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:29.945163 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 652
2026-06-20T18:43:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:30 dist-test-slave-wp6g krb5kdc[751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:30 dist-test-slave-wp6g krb5kdc[751](info): set up 2 sockets
Jun 20 18:43:30 dist-test-slave-wp6g krb5kdc[751](info): commencing operation
krb5kdc: starting...
W20260620 18:43:30.983063 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:31 dist-test-slave-wp6g krb5kdc[751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:43:31.105520 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:40723
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:40723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:31.191751 764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:31.192023 764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:31.192068 764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:31.193415 764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:31.193496 764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:31.193522 764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:31.193555 764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:31.193583 764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:31.195370 764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:40723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:40723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:31.195691 764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:31.195948 764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:43:31.201404 764 server_base.cc:1034] running on GCE node
W20260620 18:43:31.201396 769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:31.201383 770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:31.201424 772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:31.201828 764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:31.202160 764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:31.203346 764 hybrid_clock.cc:648] HybridClock initialized: now 1781981011203323 us; error 44 us; skew 500 ppm
Jun 20 18:43:31 dist-test-slave-wp6g krb5kdc[751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981011, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:31.211680 764 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:43:31.212242 764 webserver.cc:460] Webserver started at http://127.10.250.126:44761/ using document root <none> and password file <none>
I20260620 18:43:31.212512 764 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:31.212586 764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:31.212746 764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:31.213891 764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "6704648e617f45be8a4f04e031d4681f"
format_stamp: "Formatted at 2026-06-20 18:43:31 on dist-test-slave-wp6g"
server_key: "f569b35d6cfed86030b92f9fda7e00d1"
server_key_iv: "b6a8a82a567f51e31db22b426ba26954"
server_key_version: "encryptionkey@0"
I20260620 18:43:31.214284 764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "6704648e617f45be8a4f04e031d4681f"
format_stamp: "Formatted at 2026-06-20 18:43:31 on dist-test-slave-wp6g"
server_key: "f569b35d6cfed86030b92f9fda7e00d1"
server_key_iv: "b6a8a82a567f51e31db22b426ba26954"
server_key_version: "encryptionkey@0"
I20260620 18:43:31.215858 764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:43:31.216782 779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:31.216969 764 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:31.217080 764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "6704648e617f45be8a4f04e031d4681f"
format_stamp: "Formatted at 2026-06-20 18:43:31 on dist-test-slave-wp6g"
server_key: "f569b35d6cfed86030b92f9fda7e00d1"
server_key_iv: "b6a8a82a567f51e31db22b426ba26954"
server_key_version: "encryptionkey@0"
I20260620 18:43:31.217168 764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:31.229650 764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:31.232151 764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:31.232371 764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:31.236935 764 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:40723
I20260620 18:43:31.237032 831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:40723 every 8 connection(s)
I20260620 18:43:31.237397 764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:31.238370 832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:31.240793 832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f: Bootstrap starting.
I20260620 18:43:31.240993 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 764
I20260620 18:43:31.241111 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:31.241367 11241 external_mini_cluster.cc:1428] Setting key df43997746d4f24a1a9305b5f0542afb
I20260620 18:43:31.241598 832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:31.241956 832 log.cc:824] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:31.242759 832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f: No bootstrap required, opened a new log
I20260620 18:43:31.244588 832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6704648e617f45be8a4f04e031d4681f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 40723 } }
I20260620 18:43:31.244766 832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:31.244813 832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6704648e617f45be8a4f04e031d4681f, State: Initialized, Role: FOLLOWER
I20260620 18:43:31.244978 832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6704648e617f45be8a4f04e031d4681f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 40723 } }
I20260620 18:43:31.245074 832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:31.245105 832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:31.245143 832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:31.245878 832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6704648e617f45be8a4f04e031d4681f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 40723 } }
I20260620 18:43:31.246006 832 leader_election.cc:302] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6704648e617f45be8a4f04e031d4681f; no voters:
I20260620 18:43:31.246170 832 leader_election.cc:288] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:31.246207 837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:31.246428 837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [term 1 LEADER]: Becoming Leader. State: Replica: 6704648e617f45be8a4f04e031d4681f, State: Running, Role: LEADER
I20260620 18:43:31.246438 832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:31.246578 837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6704648e617f45be8a4f04e031d4681f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 40723 } }
I20260620 18:43:31.246970 839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6704648e617f45be8a4f04e031d4681f. Latest consensus state: current_term: 1 leader_uuid: "6704648e617f45be8a4f04e031d4681f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6704648e617f45be8a4f04e031d4681f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 40723 } } }
I20260620 18:43:31.246950 838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6704648e617f45be8a4f04e031d4681f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6704648e617f45be8a4f04e031d4681f" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 40723 } } }
I20260620 18:43:31.247165 838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:31.247596 835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:31.241919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56531 (local address 127.10.250.126:40723)
0620 18:43:31.242174 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:31.242183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:31.242218 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:43:31.242735 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:31.242743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:31.242812 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:31.242982 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:31.243171 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:31.244093 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:31.244960 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:31.245593 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:31.245718 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:31.246560 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:31.246563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:31.246564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:31.246773 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:31.246820 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:31.246833 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:31.246895 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:31.247029 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 18:43:31.247102 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
I20260620 18:43:31.247149 839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:31.247438 845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:31.248278 845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:31.248795 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:31.250062 845 catalog_manager.cc:1347] Generated new cluster ID: 68f5ab6f43574a9a8c627c6927a001c8
I20260620 18:43:31.250141 845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:31.253962 835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:31.250401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44711 (local address 127.10.250.126:40723)
0620 18:43:31.250444 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:31.250446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:31.250603 (+ 157us) server_negotiation.cc:407] Connection header received
0620 18:43:31.250699 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:31.250700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:31.250710 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:31.250759 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:31.250897 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:31.251595 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:31.252646 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:31.252950 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:31.253011 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:31.253742 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:31.253743 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:31.253743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:31.253748 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:31.253763 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:31.253767 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:31.253816 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:31.253863 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 18:43:31.253885 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:43:31.273689 845 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:31.274269 845 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:31.277990 845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6704648e617f45be8a4f04e031d4681f: Generated new TSK 0
I20260620 18:43:31.278251 845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:31.282490 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 764
2026-06-20T18:43:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:31 dist-test-slave-wp6g krb5kdc[863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:31 dist-test-slave-wp6g krb5kdc[863](info): set up 2 sockets
Jun 20 18:43:31 dist-test-slave-wp6g krb5kdc[863](info): commencing operation
krb5kdc: starting...
W20260620 18:43:32.326870 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:32 dist-test-slave-wp6g krb5kdc[863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:43:32.449595 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44123
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:44123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:32.533651 876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:32.533938 876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:32.533982 876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:32.535246 876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:32.535328 876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:32.535353 876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:32.535375 876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:32.535404 876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:32.536993 876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:44123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:32.537307 876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:32.537561 876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:32.542966 881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:32.543016 884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:32.542966 882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:32.543345 876 server_base.cc:1034] running on GCE node
I20260620 18:43:32.543628 876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:32.543931 876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:32.545143 876 hybrid_clock.cc:648] HybridClock initialized: now 1781981012545122 us; error 45 us; skew 500 ppm
Jun 20 18:43:32 dist-test-slave-wp6g krb5kdc[863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981012, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:32.547935 876 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:43:32.548452 876 webserver.cc:460] Webserver started at http://127.10.250.126:46419/ using document root <none> and password file <none>
I20260620 18:43:32.548696 876 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:32.548748 876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:32.548894 876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:32.550031 876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "f196162bdfe44265a75c3ac7978c1df9"
format_stamp: "Formatted at 2026-06-20 18:43:32 on dist-test-slave-wp6g"
server_key: "242cfcc5633ff2176097b6a751779eb8"
server_key_iv: "a33f593b6504d35842184ce853e1afd2"
server_key_version: "encryptionkey@0"
I20260620 18:43:32.550403 876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "f196162bdfe44265a75c3ac7978c1df9"
format_stamp: "Formatted at 2026-06-20 18:43:32 on dist-test-slave-wp6g"
server_key: "242cfcc5633ff2176097b6a751779eb8"
server_key_iv: "a33f593b6504d35842184ce853e1afd2"
server_key_version: "encryptionkey@0"
I20260620 18:43:32.552244 876 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 18:43:32.553361 891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:32.553568 876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:32.553673 876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "f196162bdfe44265a75c3ac7978c1df9"
format_stamp: "Formatted at 2026-06-20 18:43:32 on dist-test-slave-wp6g"
server_key: "242cfcc5633ff2176097b6a751779eb8"
server_key_iv: "a33f593b6504d35842184ce853e1afd2"
server_key_version: "encryptionkey@0"
I20260620 18:43:32.553823 876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:32.557238 876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:32.559676 876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:32.559798 876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:32.563642 876 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:44123
I20260620 18:43:32.563675 943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:44123 every 8 connection(s)
I20260620 18:43:32.564029 876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:32.564529 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 876
I20260620 18:43:32.564647 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:32.564849 11241 external_mini_cluster.cc:1428] Setting key 0e06d6ef4915d83d4abd9c8d7b5db492
I20260620 18:43:32.565326 944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:32.567668 947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:32.565411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54845 (local address 127.10.250.126:44123)
0620 18:43:32.565665 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:32.565687 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:32.565718 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:43:32.566241 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:32.566251 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:32.566289 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:32.566456 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:32.567201 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:32.567205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:32.567206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:32.567211 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:32.567261 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:32.567276 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:32.567360 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:32.567418 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 18:43:32.567463 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
I20260620 18:43:32.568939 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:32.569168 944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9: Bootstrap starting.
I20260620 18:43:32.569851 944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:32.570188 944 log.cc:824] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:32.571168 944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9: No bootstrap required, opened a new log
I20260620 18:43:32.572033 947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:32.570849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54509 (local address 127.10.250.126:44123)
0620 18:43:32.570902 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:32.570904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:32.571013 (+ 109us) server_negotiation.cc:407] Connection header received
0620 18:43:32.571079 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:32.571080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:32.571089 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:32.571144 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:32.571788 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:32.571790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:32.571790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:32.571794 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:32.571808 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:32.571812 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:32.571858 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:32.571927 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 18:43:32.571938 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 18:43:32.573519 944 raft_consensus.cc:348] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f196162bdfe44265a75c3ac7978c1df9" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44123 } }
I20260620 18:43:32.573705 944 raft_consensus.cc:374] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:32.573757 944 raft_consensus.cc:729] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f196162bdfe44265a75c3ac7978c1df9, State: Initialized, Role: FOLLOWER
I20260620 18:43:32.573891 944 consensus_queue.cc:260] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f196162bdfe44265a75c3ac7978c1df9" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44123 } }
I20260620 18:43:32.573992 944 raft_consensus.cc:388] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:32.574036 944 raft_consensus.cc:482] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:32.574088 944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:32.574815 944 raft_consensus.cc:504] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f196162bdfe44265a75c3ac7978c1df9" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44123 } }
I20260620 18:43:32.574939 944 leader_election.cc:302] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f196162bdfe44265a75c3ac7978c1df9; no voters:
I20260620 18:43:32.575145 944 leader_election.cc:288] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:32.575299 954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:32.575496 954 raft_consensus.cc:686] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [term 1 LEADER]: Becoming Leader. State: Replica: f196162bdfe44265a75c3ac7978c1df9, State: Running, Role: LEADER
I20260620 18:43:32.575508 944 sys_catalog.cc:564] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:32.575601 954 consensus_queue.cc:237] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f196162bdfe44265a75c3ac7978c1df9" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44123 } }
I20260620 18:43:32.576027 955 sys_catalog.cc:455] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f196162bdfe44265a75c3ac7978c1df9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f196162bdfe44265a75c3ac7978c1df9" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44123 } } }
I20260620 18:43:32.576037 956 sys_catalog.cc:455] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f196162bdfe44265a75c3ac7978c1df9. Latest consensus state: current_term: 1 leader_uuid: "f196162bdfe44265a75c3ac7978c1df9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f196162bdfe44265a75c3ac7978c1df9" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44123 } } }
I20260620 18:43:32.576167 955 sys_catalog.cc:458] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:32.576175 956 sys_catalog.cc:458] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:32.576418 963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:32.577005 963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:32.581975 963 catalog_manager.cc:1347] Generated new cluster ID: b020b726266645f1b7828d6ef4f33137
I20260620 18:43:32.582049 963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:32.608915 963 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:32.609140 963 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:32.616860 963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f196162bdfe44265a75c3ac7978c1df9: Generated new TSK 0
I20260620 18:43:32.617097 963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:32.626369 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 876
2026-06-20T18:43:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1343 ms)
[----------] 18 tests from AuthTokenIssuingTest (24333 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:32 dist-test-slave-wp6g krb5kdc[975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:32 dist-test-slave-wp6g krb5kdc[975](info): set up 2 sockets
Jun 20 18:43:32 dist-test-slave-wp6g krb5kdc[975](info): commencing operation
krb5kdc: starting...
W20260620 18:43:33.702955 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:33 dist-test-slave-wp6g krb5kdc[975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:43:33.829109 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44271
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:36005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:44271,127.10.250.125:46803,127.10.250.124:41005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:33.912232 988 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:43:33.912503 988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:33.912535 988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:33.912576 988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:33.914108 988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:33.914214 988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:33.914243 988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:33.914268 988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:33.914296 988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:33.915936 988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:44271,127.10.250.125:46803,127.10.250.124:41005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:44271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:33.916231 988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:33.916467 988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:33.921537 994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:33.921577 988 server_base.cc:1034] running on GCE node
W20260620 18:43:33.921527 996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:33.921527 993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:33.921988 988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:33.922295 988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:33.923460 988 hybrid_clock.cc:648] HybridClock initialized: now 1781981013923429 us; error 53 us; skew 500 ppm
Jun 20 18:43:33 dist-test-slave-wp6g krb5kdc[975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981013, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:33.925930 988 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:43:33.926427 988 webserver.cc:460] Webserver started at http://127.10.250.126:37021/ using document root <none> and password file <none>
I20260620 18:43:33.926695 988 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:33.926766 988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:33.926908 988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:33.927987 988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "75ab6595db614ab9a0cb146db251accb"
format_stamp: "Formatted at 2026-06-20 18:43:33 on dist-test-slave-wp6g"
server_key: "5d12f0cf739e7821f0437a9c0e5f6f9b"
server_key_iv: "8c504df879446913cd42ceb1ff5903bb"
server_key_version: "encryptionkey@0"
I20260620 18:43:33.928349 988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "75ab6595db614ab9a0cb146db251accb"
format_stamp: "Formatted at 2026-06-20 18:43:33 on dist-test-slave-wp6g"
server_key: "5d12f0cf739e7821f0437a9c0e5f6f9b"
server_key_iv: "8c504df879446913cd42ceb1ff5903bb"
server_key_version: "encryptionkey@0"
I20260620 18:43:33.929751 988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:43:33.930637 1003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:33.930814 988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:33.930918 988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "75ab6595db614ab9a0cb146db251accb"
format_stamp: "Formatted at 2026-06-20 18:43:33 on dist-test-slave-wp6g"
server_key: "5d12f0cf739e7821f0437a9c0e5f6f9b"
server_key_iv: "8c504df879446913cd42ceb1ff5903bb"
server_key_version: "encryptionkey@0"
I20260620 18:43:33.930999 988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:33.939849 988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:33.941968 988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:33.942138 988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:33.946640 988 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:44271
I20260620 18:43:33.946692 1055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:44271 every 8 connection(s)
I20260620 18:43:33.947068 988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:33.947937 1056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:33.949512 1056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:43:33.954600 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 988
I20260620 18:43:33.954735 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:33.954921 11241 external_mini_cluster.cc:1428] Setting key 7738dae559b4520bda6950b6247545b1
Jun 20 18:43:33 dist-test-slave-wp6g krb5kdc[975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981013, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:33.959973 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:33.950294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.10.250.126:44271)
0620 18:43:33.950499 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:33.950503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:33.950634 (+ 131us) server_negotiation.cc:407] Connection header received
0620 18:43:33.951167 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:33.951169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:33.951187 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:33.951270 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:33.951850 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:33.952657 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:33.953531 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:33.953884 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:33.953981 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:33.956459 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:33.956466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:33.956466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:33.956472 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:33.958238 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:33.958741 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:33.958744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:33.958744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:33.958824 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:33.959125 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:33.959126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:33.959126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:33.959338 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:33.959398 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:33.959656 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 18:43:33.959785 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":169,"thread_start_us":127,"threads_started":1}
I20260620 18:43:33.960054 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:33.950262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44271 (local address 127.0.0.1:37792)
0620 18:43:33.950448 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:33.950493 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:33.951030 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:33.951316 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:33.951319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:33.951598 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:33.951754 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:33.951758 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:33.952755 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:33.952756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:33.953438 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:33.953439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:33.953984 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:33.953985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:33.954059 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:33.954734 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:33.954748 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:33.956366 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:33.958326 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:33.958335 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:33.958337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:33.958662 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:33.958897 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:33.958898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:33.958898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:33.959017 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:33.959410 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:33.959417 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:33.959504 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 18:43:33.959628 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 18:43:33.959879 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":56,"thread_start_us":71,"threads_started":1}
I20260620 18:43:33.960807 1056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:43:33.961211 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:33.961072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:46803 (local address 127.0.0.1:55602)
0620 18:43:33.961129 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:33.961183 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 18:43:33.961380 1005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.250.125:46803: connect: Connection refused (error 111)
I20260620 18:43:33.961870 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:33.961756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:46803 (local address 127.0.0.1:55616)
0620 18:43:33.961808 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:33.961842 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 18:43:33.962026 1056 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.125:46803: Network error: Client connection negotiation failed: client connection to 127.10.250.125:46803: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.10.250.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.125.keytab.
Entry for principal kudu/127.10.250.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.125.keytab.
I20260620 18:43:33.981004 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.125:46803
--webserver_interface=127.10.250.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.125.keytab
--principal=kudu/127.10.250.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:36005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:44271,127.10.250.125:46803,127.10.250.124:41005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:43:34.011334 1056 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } attempt: 1
I20260620 18:43:34.011878 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.011704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:46803 (local address 127.0.0.1:55622)
0620 18:43:34.011784 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.011836 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 18:43:34.012326 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.012229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:46803 (local address 127.0.0.1:55630)
0620 18:43:34.012277 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.012306 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 18:43:34.012465 1056 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.125:46803: Network error: Client connection negotiation failed: client connection to 127.10.250.125:46803: connect: Connection refused (error 111)
W20260620 18:43:34.065228 1063 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:43:34.065448 1063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:34.065466 1063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:34.065486 1063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:34.066701 1063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:34.066743 1063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:34.066757 1063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:34.066771 1063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:34.066782 1063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:34.068324 1063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:44271,127.10.250.125:46803,127.10.250.124:41005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.125.keytab
--principal=kudu/127.10.250.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.125:46803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.250.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:34.068580 1063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:34.068806 1063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:34.074214 1069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:34.074271 1068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:34.074311 1063 server_base.cc:1034] running on GCE node
W20260620 18:43:34.074271 1071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:34.074697 1063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:34.075018 1063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:34.076174 1063 hybrid_clock.cc:648] HybridClock initialized: now 1781981014076164 us; error 32 us; skew 500 ppm
Jun 20 18:43:34 dist-test-slave-wp6g krb5kdc[975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981014, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:34.078979 1063 init.cc:377] Logged in from keytab as kudu/127.10.250.125@KRBTEST.COM (short username kudu)
I20260620 18:43:34.079460 1063 webserver.cc:460] Webserver started at http://127.10.250.125:39817/ using document root <none> and password file <none>
I20260620 18:43:34.079708 1063 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:34.079754 1063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:34.079890 1063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:34.080678 1056 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } attempt: 2
I20260620 18:43:34.081072 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.080945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:46803 (local address 127.0.0.1:55646)
0620 18:43:34.081009 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.081043 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 18:43:34.081054 1063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/data/instance:
uuid: "7806f1d33c4340ceacec95b30315821c"
format_stamp: "Formatted at 2026-06-20 18:43:34 on dist-test-slave-wp6g"
server_key: "a047c983b845139986d9155b7eda638f"
server_key_iv: "40a1ee70aef41138133f64cb5918b3b7"
server_key_version: "encryptionkey@0"
I20260620 18:43:34.081459 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.081324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:46803 (local address 127.0.0.1:55654)
0620 18:43:34.081386 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.081422 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 18:43:34.081506 1063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/wal/instance:
uuid: "7806f1d33c4340ceacec95b30315821c"
format_stamp: "Formatted at 2026-06-20 18:43:34 on dist-test-slave-wp6g"
server_key: "a047c983b845139986d9155b7eda638f"
server_key_iv: "40a1ee70aef41138133f64cb5918b3b7"
server_key_version: "encryptionkey@0"
W20260620 18:43:34.081583 1056 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.125:46803: Network error: Client connection negotiation failed: client connection to 127.10.250.125:46803: connect: Connection refused (error 111)
I20260620 18:43:34.082907 1063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:43:34.083755 1078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:34.083951 1063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:43:34.084045 1063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/wal
uuid: "7806f1d33c4340ceacec95b30315821c"
format_stamp: "Formatted at 2026-06-20 18:43:34 on dist-test-slave-wp6g"
server_key: "a047c983b845139986d9155b7eda638f"
server_key_iv: "40a1ee70aef41138133f64cb5918b3b7"
server_key_version: "encryptionkey@0"
I20260620 18:43:34.084147 1063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:34.101066 1063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:34.103159 1063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:34.103302 1063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:34.107558 1063 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.125:46803
I20260620 18:43:34.107656 1130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.125:46803 every 8 connection(s)
I20260620 18:43:34.107978 1063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/data/info.pb
I20260620 18:43:34.108847 1131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:34.110414 1131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:34 dist-test-slave-wp6g krb5kdc[975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981014, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.125@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:34.116444 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 1063
I20260620 18:43:34.116533 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-1/wal/instance
I20260620 18:43:34.116714 11241 external_mini_cluster.cc:1428] Setting key 8a6de3a9926f39b3acf33f7154f049a5
I20260620 18:43:34.120070 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.111189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.10.250.126:44271)
0620 18:43:34.111283 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.111285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.111483 (+ 198us) server_negotiation.cc:407] Connection header received
0620 18:43:34.111959 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.111961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.111972 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.112031 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.112630 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.113127 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.114186 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.114418 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.114477 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.116772 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:34.116775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:34.116776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:34.116787 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:34.118290 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.118853 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.118854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.118854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.118918 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.119271 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.119272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.119273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.119412 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:34.119473 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:34.119857 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 18:43:34.119976 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:43:34.120237 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.111169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44271 (local address 127.0.0.1:37798)
0620 18:43:34.111346 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.111392 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:34.111876 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:34.112091 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:34.112094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:34.112343 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:34.112553 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:34.112557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.113245 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.113246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:34.114094 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:34.114097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.114500 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.114501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:34.114633 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.115128 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:34.115137 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:34.116649 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:34.118397 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:34.118406 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:34.118409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:34.118768 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:34.119016 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:34.119017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:34.119017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:34.119163 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:34.119491 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:34.119497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:34.119675 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 18:43:34.119823 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 18:43:34.119990 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260620 18:43:34.120904 1131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.250.124@KRBTEST.COM; defaulting to no policy
Jun 20 18:43:34 dist-test-slave-wp6g krb5kdc[975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981014, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.125@KRBTEST.COM for kudu/127.10.250.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.124@KRBTEST.COM" created.
I20260620 18:43:34.128738 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.121275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55660 (local address 127.10.250.125:46803)
0620 18:43:34.121445 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.121449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.121464 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:43:34.121481 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.121483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.121493 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.121577 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.121852 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.122644 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.123363 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.123677 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.123775 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.125777 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:34.125785 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:34.125785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:34.125792 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:34.127305 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.127729 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.127732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.127733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.127798 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.128033 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.128034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.128034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.128241 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:34.128312 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:34.128486 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 18:43:34.128620 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":146,"thread_start_us":103,"threads_started":1}
I20260620 18:43:34.128737 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.121197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:46803 (local address 127.0.0.1:55660)
0620 18:43:34.121257 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.121267 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:34.121308 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:34.121577 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:34.121578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:34.121724 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:34.121770 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:34.121771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.122752 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.122753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:34.123264 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:34.123265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.123773 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.123774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:34.123839 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.124330 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:34.124335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:34.125695 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:34.127384 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:34.127385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:34.127386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:34.127570 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:34.127907 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:34.127908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:34.127908 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:34.127965 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:34.128317 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:34.128318 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:34.128373 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 18:43:34.128451 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:43:34.128570 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 18:43:34.129523 1131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:43:34.129905 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.129793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41005 (local address 127.0.0.1:59796)
0620 18:43:34.129842 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.129885 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 18:43:34.130067 1080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41005: connect: Connection refused (error 111)
I20260620 18:43:34.130645 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.130508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41005 (local address 127.0.0.1:59802)
0620 18:43:34.130604 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.130627 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260620 18:43:34.130776 1131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.124:41005: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41005: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.124.keytab.
Entry for principal kudu/127.10.250.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.124.keytab.
I20260620 18:43:34.143652 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.124:41005
--webserver_interface=127.10.250.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.124.keytab
--principal=kudu/127.10.250.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:36005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:44271,127.10.250.125:46803,127.10.250.124:41005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:43:34.172760 1056 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } attempt: 3
Jun 20 18:43:34 dist-test-slave-wp6g krb5kdc[975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981013, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for kudu/127.10.250.125@KRBTEST.COM
I20260620 18:43:34.180037 1131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } attempt: 1
I20260620 18:43:34.180466 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.180331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41005 (local address 127.0.0.1:59816)
0620 18:43:34.180393 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.180429 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 18:43:34.180940 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.180806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41005 (local address 127.0.0.1:59828)
0620 18:43:34.180864 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.180903 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 18:43:34.180968 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.173219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:46803 (local address 127.0.0.1:55674)
0620 18:43:34.173277 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.173297 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:34.173377 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:34.173532 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:34.173533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:34.173713 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:34.173780 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:34.173782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.174470 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.174471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:34.175070 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:34.175072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.175520 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.175521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:34.175597 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.176167 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:34.176173 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:34.177728 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:34.179541 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:34.179543 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:34.179543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:34.179751 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:34.179988 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:34.179989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:34.179990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:34.180058 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:34.180583 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:34.180585 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:34.180643 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 18:43:34.180741 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:43:34.180866 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 18:43:34.181037 1131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.124:41005: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41005: connect: Connection refused (error 111)
I20260620 18:43:34.181237 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.173315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55674 (local address 127.10.250.125:46803)
0620 18:43:34.173391 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.173394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.173405 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:43:34.173445 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.173446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.173464 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.173525 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.173871 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.174375 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.175196 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.175414 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.175522 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.177845 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:34.177847 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:34.177847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:34.177855 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:34.179443 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.179841 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.179845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.179845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.179905 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.180134 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.180135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.180135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.180307 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:34.180395 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:34.180880 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 18:43:34.180962 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:43:34.181694 1056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:43:34.182031 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.181932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41005 (local address 127.0.0.1:59842)
0620 18:43:34.181978 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.182006 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 18:43:34.182325 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.182234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41005 (local address 127.0.0.1:59844)
0620 18:43:34.182283 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.182305 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 18:43:34.182416 1056 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.124:41005: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41005: connect: Connection refused (error 111)
I20260620 18:43:34.213568 1056 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } attempt: 1
I20260620 18:43:34.214040 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.213906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41005 (local address 127.0.0.1:59860)
0620 18:43:34.213971 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.214011 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 18:43:34.214368 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.214274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41005 (local address 127.0.0.1:59866)
0620 18:43:34.214320 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.214343 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 18:43:34.214458 1056 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.124:41005: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41005: connect: Connection refused (error 111)
W20260620 18:43:34.228811 1138 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:43:34.229040 1138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:34.229059 1138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:34.229127 1138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:34.230362 1138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:34.230410 1138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:34.230425 1138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:34.230438 1138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:34.230574 1138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:34.232223 1138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:36005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:44271,127.10.250.125:46803,127.10.250.124:41005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.124.keytab
--principal=kudu/127.10.250.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.124:41005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.250.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:34.232436 1138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:34.232630 1138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:34.237852 1143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:34.237833 1144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:34.237833 1146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:34.237897 1138 server_base.cc:1034] running on GCE node
I20260620 18:43:34.238368 1138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:34.238729 1138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:34.239879 1138 hybrid_clock.cc:648] HybridClock initialized: now 1781981014239852 us; error 41 us; skew 500 ppm
Jun 20 18:43:34 dist-test-slave-wp6g krb5kdc[975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981014, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:34.242470 1138 init.cc:377] Logged in from keytab as kudu/127.10.250.124@KRBTEST.COM (short username kudu)
I20260620 18:43:34.242972 1138 webserver.cc:460] Webserver started at http://127.10.250.124:38775/ using document root <none> and password file <none>
I20260620 18:43:34.243227 1138 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:34.243299 1138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:34.243446 1138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:34.244522 1138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/data/instance:
uuid: "67e602ef205a46e08b544c32fa308799"
format_stamp: "Formatted at 2026-06-20 18:43:34 on dist-test-slave-wp6g"
server_key: "1f823fb73ac5243e1938addabc50fbfd"
server_key_iv: "e106d854c580ef6d7355837d891b5917"
server_key_version: "encryptionkey@0"
I20260620 18:43:34.244889 1138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/wal/instance:
uuid: "67e602ef205a46e08b544c32fa308799"
format_stamp: "Formatted at 2026-06-20 18:43:34 on dist-test-slave-wp6g"
server_key: "1f823fb73ac5243e1938addabc50fbfd"
server_key_iv: "e106d854c580ef6d7355837d891b5917"
server_key_version: "encryptionkey@0"
I20260620 18:43:34.246251 1138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260620 18:43:34.247061 1153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:34.247236 1138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:34.247334 1138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/wal
uuid: "67e602ef205a46e08b544c32fa308799"
format_stamp: "Formatted at 2026-06-20 18:43:34 on dist-test-slave-wp6g"
server_key: "1f823fb73ac5243e1938addabc50fbfd"
server_key_iv: "e106d854c580ef6d7355837d891b5917"
server_key_version: "encryptionkey@0"
I20260620 18:43:34.247417 1138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:34.249179 1131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } attempt: 2
I20260620 18:43:34.249650 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.249514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41005 (local address 127.0.0.1:59868)
0620 18:43:34.249576 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.249617 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 18:43:34.249960 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.249882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41005 (local address 127.0.0.1:59882)
0620 18:43:34.249910 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.249938 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260620 18:43:34.250051 1131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.124:41005: Network error: Client connection negotiation failed: client connection to 127.10.250.124:41005: connect: Connection refused (error 111)
I20260620 18:43:34.272276 1138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:34.274427 1138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:34.274638 1138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:34.278497 1138 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.124:41005
I20260620 18:43:34.278580 1205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.124:41005 every 8 connection(s)
I20260620 18:43:34.279021 1138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/data/info.pb
I20260620 18:43:34.279904 1206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:34.281445 1206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:34 dist-test-slave-wp6g krb5kdc[975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981014, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.124@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:34.289162 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 1138
I20260620 18:43:34.289291 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980738687848-11241-0/minicluster-data/master-2/wal/instance
I20260620 18:43:34.289479 11241 external_mini_cluster.cc:1428] Setting key 35a8159d10ef0e14331287f0967ad1d7
I20260620 18:43:34.289613 1056 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } attempt: 2
I20260620 18:43:34.291107 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.282147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.10.250.126:44271)
0620 18:43:34.282265 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.282267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.282398 (+ 131us) server_negotiation.cc:407] Connection header received
0620 18:43:34.282931 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.282932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.282941 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.282993 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.283580 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.284055 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.285046 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.285246 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.285301 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.287636 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:34.287637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:34.287638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:34.287644 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:34.289061 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.289844 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.289845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.289846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.289911 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.290236 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.290237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.290237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.290403 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:34.290448 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:34.290923 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 18:43:34.291008 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 18:43:34.291327 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.282106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:44271 (local address 127.0.0.1:37802)
0620 18:43:34.282279 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.282315 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:34.282844 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:34.283047 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:34.283050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:34.283312 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:34.283506 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:34.283510 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.284160 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.284161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:34.284970 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:34.284972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.285341 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.285342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:34.285451 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.286030 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:34.286038 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:34.287544 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:34.289207 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:34.289217 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:34.289220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:34.289518 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:34.289996 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:34.289997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:34.289997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:34.290153 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:34.290500 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:34.290506 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:34.290739 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 18:43:34.290870 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 18:43:34.291052 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260620 18:43:34.292178 1206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:34 dist-test-slave-wp6g krb5kdc[975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981013, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for kudu/127.10.250.124@KRBTEST.COM
Jun 20 18:43:34 dist-test-slave-wp6g krb5kdc[975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.124@KRBTEST.COM
Jun 20 18:43:34 dist-test-slave-wp6g krb5kdc[975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981014, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.124@KRBTEST.COM for kudu/127.10.250.125@KRBTEST.COM
I20260620 18:43:34.300033 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.289886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41005 (local address 127.0.0.1:59896)
0620 18:43:34.289938 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.289950 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:34.289983 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:34.290375 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:34.290376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:34.290562 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:34.290605 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:34.290607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.291784 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.291785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:34.292586 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:34.292591 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.293222 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.293223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:34.293285 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.293772 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:34.293777 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:34.295414 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:34.298734 (+ 3320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:34.298736 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:34.298736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:34.298928 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:34.299254 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:34.299255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:34.299255 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:34.299325 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:34.299654 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:34.299655 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:34.299716 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 18:43:34.299805 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 18:43:34.299926 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 18:43:34.300046 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.290045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.10.250.124:41005)
0620 18:43:34.290202 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.290208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.290232 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:43:34.290257 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.290259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.290275 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.290361 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.290672 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.291657 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.292662 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.293137 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.293211 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.295506 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:34.295518 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:34.295519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:34.295527 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:34.298642 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.299033 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.299038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.299039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.299112 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.299395 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.299396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.299397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.299573 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:34.299669 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:34.299841 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 18:43:34.299944 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":131,"thread_start_us":56,"threads_started":1}
I20260620 18:43:34.300500 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.292540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55678 (local address 127.10.250.125:46803)
0620 18:43:34.292582 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.292585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.292593 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:43:34.292619 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.292620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.292631 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.292680 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.292999 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.293624 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.294881 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.295210 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.295275 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.297513 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:34.297514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:34.297514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:34.297523 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:34.299042 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.299564 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.299565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.299566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.299630 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.299845 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.299846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.299846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.299991 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:34.300033 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:34.300279 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 18:43:34.300384 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:43:34.300848 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.292436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:46803 (local address 127.0.0.1:55678)
0620 18:43:34.292491 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.292503 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:34.292546 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:34.292717 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:34.292718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:34.292852 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:34.292900 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:34.292901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.293713 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.293714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:34.294480 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:34.294482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.295328 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.295330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:34.295407 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.295805 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:34.295810 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:34.297421 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:34.299150 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:34.299151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:34.299152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:34.299467 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:34.299711 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:34.299712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:34.299712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:34.299771 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:34.300095 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:34.300096 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:34.300139 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 18:43:34.300636 (+ 497us) client_negotiation.cc:240] Negotiation successful
0620 18:43:34.300742 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 18:43:34.301236 1206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:43:34.301597 1210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.289927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.10.250.124:41005)
0620 18:43:34.290202 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.290208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.290232 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:43:34.290257 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.290259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.290275 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.290361 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.291456 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.292198 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.293270 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.293551 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.293629 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.296910 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:34.296911 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:34.296912 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:34.296918 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:34.299926 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.300356 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.300357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.300358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.300415 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.300639 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.300640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.300640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.300759 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:34.300912 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:34.301368 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 18:43:34.301461 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":241,"thread_start_us":81,"threads_started":1}
I20260620 18:43:34.302584 1056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb: Bootstrap starting.
I20260620 18:43:34.303392 1056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:34.303704 1056 log.cc:824] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:34.304513 1056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb: No bootstrap required, opened a new log
Jun 20 18:43:34 dist-test-slave-wp6g krb5kdc[975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981014, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.124@KRBTEST.COM for kudu/127.10.250.124@KRBTEST.COM
I20260620 18:43:34.306983 1056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } }
I20260620 18:43:34.307137 1056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ab6595db614ab9a0cb146db251accb, State: Initialized, Role: FOLLOWER
I20260620 18:43:34.307219 1056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } }
Jun 20 18:43:34 dist-test-slave-wp6g krb5kdc[975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:34.307574 1056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:34.307531 1213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } } }
I20260620 18:43:34.307636 1213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:34.308744 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.301508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41005 (local address 127.0.0.1:59898)
0620 18:43:34.301546 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.301556 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:34.301592 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:34.301774 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:34.301775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:34.301919 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:34.301956 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:34.301957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.302794 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.302795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:34.303423 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:34.303425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.303859 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.303860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:34.303900 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.304566 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:34.304572 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:34.305859 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:34.307555 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:34.307556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:34.307556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:34.307758 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:34.307964 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:34.307965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:34.307965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:34.308040 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:34.308434 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:34.308435 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:34.308471 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:43:34.308513 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 18:43:34.308611 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 18:43:34.309113 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.301574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59898 (local address 127.10.250.124:41005)
0620 18:43:34.301613 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.301615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.301626 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:43:34.301648 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.301649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.301661 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.301727 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.302031 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.302725 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.303530 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.303742 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.303816 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.305959 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:34.305960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:34.305960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:34.305967 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:34.307449 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.307836 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.307837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.307837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.307880 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.308123 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.308124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.308125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.308342 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:34.308868 (+ 526us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:34.308914 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 18:43:34.308999 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 18:43:34.310698 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.302310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.10.250.126:44271)
0620 18:43:34.302348 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.302350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.302358 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:43:34.302403 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.302404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.302413 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.302462 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.303077 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.303765 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.304762 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.305027 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.305068 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.307788 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:34.307789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:34.307790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:34.307798 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:34.309378 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.309813 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.309815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.309815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.309869 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.310074 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.310075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.310075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.310220 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:34.310349 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:34.310487 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 18:43:34.310604 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:43:34.310788 1206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799: Bootstrap starting.
I20260620 18:43:34.311538 1206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:34.311859 1206 log.cc:824] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799: Log is configured to *not* fsync() on all Append() calls
W20260620 18:43:34.312316 1224 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:43:34.312415 1224 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:43:34.312664 1206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799: No bootstrap required, opened a new log
I20260620 18:43:34.310873 988 master_runner.cc:186] Error getting master registration for 127.10.250.125:46803: OK, instance_id { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" instance_seqno: 1781981014106291 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:43:34.313915 988 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:43:34.314764 1206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } }
I20260620 18:43:34.314950 1206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e602ef205a46e08b544c32fa308799, State: Initialized, Role: FOLLOWER
I20260620 18:43:34.315048 1206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } }
I20260620 18:43:34.315372 1206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:34.315335 1227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } } }
I20260620 18:43:34.315559 1227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 18:43:34 dist-test-slave-wp6g krb5kdc[975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.125@KRBTEST.COM
W20260620 18:43:34.317646 1238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:43:34.317715 1238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:43:34.321774 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.312158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.10.250.125:46803)
0620 18:43:34.312230 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.312233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.312242 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:43:34.312269 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.312270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.312282 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.312346 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.313014 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.313717 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.314480 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.314776 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.314869 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.318303 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:34.318305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:34.318306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:34.318314 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:34.320323 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.320759 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.320760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.320761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.320824 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.321038 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.321039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.321039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.321195 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:34.321349 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:34.321584 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 18:43:34.321678 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:43:34.319629 1138 master_runner.cc:186] Error getting master registration for 127.10.250.125:46803: OK, instance_id { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" instance_seqno: 1781981014106291 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:43:34.322067 1138 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:43:34.324023 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:34.325016 1030 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "75ab6595db614ab9a0cb146db251accb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37818
I20260620 18:43:34.325174 1030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 18:43:34.325225 1030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:34.326186 1030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } }
I20260620 18:43:34.326442 1030 leader_election.cc:288] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [CANDIDATE]: Term 1 election: Requested vote from peers 7806f1d33c4340ceacec95b30315821c (127.10.250.125:46803), 67e602ef205a46e08b544c32fa308799 (127.10.250.124:41005)
I20260620 18:43:34.326946 1180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75ab6595db614ab9a0cb146db251accb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "67e602ef205a46e08b544c32fa308799"
I20260620 18:43:34.327080 1180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:34.327970 1180 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75ab6595db614ab9a0cb146db251accb in term 1.
I20260620 18:43:34.328243 1005 leader_election.cc:302] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e602ef205a46e08b544c32fa308799, 75ab6595db614ab9a0cb146db251accb; no voters:
I20260620 18:43:34.328400 1213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:34.328552 1213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [term 1 LEADER]: Becoming Leader. State: Replica: 75ab6595db614ab9a0cb146db251accb, State: Running, Role: LEADER
I20260620 18:43:34.328639 1213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } }
I20260620 18:43:34.329249 1244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75ab6595db614ab9a0cb146db251accb. Latest consensus state: current_term: 1 leader_uuid: "75ab6595db614ab9a0cb146db251accb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } } }
I20260620 18:43:34.329375 1244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:34.331483 1246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:34.331761 1246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:34.333817 1180 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799 [term 1 FOLLOWER]: Refusing update from remote peer 75ab6595db614ab9a0cb146db251accb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:34.334190 1244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:34.336136 1227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75ab6595db614ab9a0cb146db251accb. Latest consensus state: current_term: 1 leader_uuid: "75ab6595db614ab9a0cb146db251accb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } } }
I20260620 18:43:34.336242 1227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:34.341557 1131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } attempt: 3
I20260620 18:43:34.345587 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.330888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.10.250.125:46803)
0620 18:43:34.330938 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.330941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.335437 (+ 4496us) server_negotiation.cc:407] Connection header received
0620 18:43:34.335518 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.335520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.335533 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.335603 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.336875 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.337729 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.338907 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.339291 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.339368 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.340949 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:34.340951 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:34.340951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:34.340960 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:34.343250 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.344166 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.344168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.344168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.344246 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.344624 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.344625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.344626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.344781 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:34.344958 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:34.345373 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 18:43:34.345464 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 18:43:34 dist-test-slave-wp6g krb5kdc[975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981014, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.125@KRBTEST.COM for kudu/127.10.250.124@KRBTEST.COM
I20260620 18:43:34.348022 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.331100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59902 (local address 127.10.250.124:41005)
0620 18:43:34.331148 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.331150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.338680 (+ 7530us) server_negotiation.cc:407] Connection header received
0620 18:43:34.338783 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.338784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.338796 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.338828 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.339476 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.340109 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.341114 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.341442 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.341504 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.344906 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:34.344908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:34.344908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:34.344918 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:34.346641 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.347081 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.347082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.347083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.347144 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.347365 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.347366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.347366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.347499 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:34.347611 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:34.347806 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 18:43:34.347914 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:43:34.350296 1210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.342033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.10.250.124:41005)
0620 18:43:34.342075 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.342077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.342083 (+ 6us) server_negotiation.cc:407] Connection header received
0620 18:43:34.342103 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.342104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.342126 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.342191 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.342594 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.343283 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.344297 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.344622 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.344677 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.347282 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:34.347284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:34.347284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:34.347292 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:34.348963 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.349400 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.349401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.349402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.349456 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.349655 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.349656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.349657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.349847 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:34.349929 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:34.350106 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 18:43:34.350194 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:43:34.350657 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.341825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:41005 (local address 127.0.0.1:59916)
0620 18:43:34.341911 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:34.341921 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:34.341957 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:34.342276 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:34.342277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:34.342423 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:34.342463 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:34.342465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.343386 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.343387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:34.344210 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:34.344213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.344729 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.344730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:34.344790 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.345423 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:34.345428 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:34.347182 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:34.349048 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:34.349049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:34.349050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:34.349314 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:34.349521 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:34.349522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:34.349523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:34.349576 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:34.349935 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:34.349936 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:34.349987 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 18:43:34.350061 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 18:43:34.350255 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 18:43:34.352674 1131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c: Bootstrap starting.
I20260620 18:43:34.353426 1131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:34.353775 1131 log.cc:824] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:34.354743 1131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c: No bootstrap required, opened a new log
I20260620 18:43:34.355991 1244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75ab6595db614ab9a0cb146db251accb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } } }
I20260620 18:43:34.356108 1244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:34.356701 1227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75ab6595db614ab9a0cb146db251accb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } } }
I20260620 18:43:34.356803 1227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:34.356802 1131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } }
I20260620 18:43:34.356945 1131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7806f1d33c4340ceacec95b30315821c, State: Initialized, Role: FOLLOWER
I20260620 18:43:34.357019 1131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } }
I20260620 18:43:34.357131 1251 mvcc.cc:205] Tried to move back new op lower bound from 7298994234707750912 to 7298994234690850816. Current Snapshot: MvccSnapshot[applied={T|T < 7298994234707750912}]
I20260620 18:43:34.357235 1250 mvcc.cc:205] Tried to move back new op lower bound from 7298994234707750912 to 7298994234690850816. Current Snapshot: MvccSnapshot[applied={T|T < 7298994234707750912}]
I20260620 18:43:34.357327 1131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:34.357867 1246 catalog_manager.cc:1347] Generated new cluster ID: bad4b00f0faa48c8aca6de4ae53fef6d
I20260620 18:43:34.357914 1246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:34.358680 1104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:34.359591 1104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c [term 1 FOLLOWER]: Refusing update from remote peer 75ab6595db614ab9a0cb146db251accb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:34.359782 1105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75ab6595db614ab9a0cb146db251accb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7806f1d33c4340ceacec95b30315821c"
I20260620 18:43:34.360163 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:34.331349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.10.250.126:44271)
0620 18:43:34.331387 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.331389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.341215 (+ 9826us) server_negotiation.cc:407] Connection header received
0620 18:43:34.341306 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.341307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.341318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.341347 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.342257 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.343026 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.354638 (+ 11612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:34.354931 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:34.355009 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:34.357329 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:34.357330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:34.357330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:34.357338 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:34.358999 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.359367 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.359368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.359368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.359414 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:34.359583 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:34.359583 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:34.359584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:34.359707 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:34.359783 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:34.359978 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 18:43:34.360079 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:43:34.360589 1105 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75ab6595db614ab9a0cb146db251accb in term 1.
I20260620 18:43:34.361207 1213 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:34.362905 1269 mvcc.cc:205] Tried to move back new op lower bound from 7298994234707750912 to 7298994234690850816. Current Snapshot: MvccSnapshot[applied={T|T < 7298994234707750912}]
I20260620 18:43:34.366634 1256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75ab6595db614ab9a0cb146db251accb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } } }
I20260620 18:43:34.366855 1256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:34.366916 1256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75ab6595db614ab9a0cb146db251accb. Latest consensus state: current_term: 1 leader_uuid: "75ab6595db614ab9a0cb146db251accb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } } }
I20260620 18:43:34.366958 1256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:34.366981 1256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75ab6595db614ab9a0cb146db251accb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } } }
I20260620 18:43:34.367017 1256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c [sys.catalog]: This master's current role is: FOLLOWER
W20260620 18:43:34.367815 1267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:43:34.367883 1267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:43:34.373232 1213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75ab6595db614ab9a0cb146db251accb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ab6595db614ab9a0cb146db251accb" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 44271 } } peers { permanent_uuid: "7806f1d33c4340ceacec95b30315821c" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 46803 } } peers { permanent_uuid: "67e602ef205a46e08b544c32fa308799" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 41005 } } }
I20260620 18:43:34.373364 1213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:34.373458 1246 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:34.374207 1246 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:34.389612 1246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75ab6595db614ab9a0cb146db251accb: Generated new TSK 0
I20260620 18:43:34.389891 1246 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:43:34.420410 1135 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:34.418713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.10.250.125:46803)
0620 18:43:34.418766 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.418768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.419520 (+ 752us) server_negotiation.cc:407] Connection header received
0620 18:43:34.419731 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.419733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.419746 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.419778 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.420316 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55706: BlockingRecv error: recv got EOF from 127.0.0.1:55706 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 18:43:34.421157 1059 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:34.418408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37840 (local address 127.10.250.126:44271)
0620 18:43:34.418461 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.418464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.419673 (+ 1209us) server_negotiation.cc:407] Connection header received
0620 18:43:34.420664 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.420665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.420677 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.420707 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.421087 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37840: BlockingRecv error: recv got EOF from 127.0.0.1:37840 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 18:43:34.422003 1210 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:34.419130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.10.250.124:41005)
0620 18:43:34.419172 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.419174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.421418 (+ 2244us) server_negotiation.cc:407] Connection header received
0620 18:43:34.421587 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.421588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.421599 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.421630 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.421943 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59930: BlockingRecv error: recv got EOF from 127.0.0.1:59930 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 18:43:34.424669 1135 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:34.423759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.10.250.125:46803)
0620 18:43:34.423802 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.423804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.424088 (+ 284us) server_negotiation.cc:407] Connection header received
0620 18:43:34.424209 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.424210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.424232 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.424261 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.424614 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55714: BlockingRecv error: recv got EOF from 127.0.0.1:55714 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 18:43:34.428323 1135 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:34.427304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.10.250.125:46803)
0620 18:43:34.427345 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.427346 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.427621 (+ 275us) server_negotiation.cc:407] Connection header received
0620 18:43:34.427812 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.427813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.427825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.427856 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.428264 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55726: BlockingRecv error: recv got EOF from 127.0.0.1:55726 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 18:43:34.432530 1135 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:34.431685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.10.250.125:46803)
0620 18:43:34.431727 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.431729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.431972 (+ 243us) server_negotiation.cc:407] Connection header received
0620 18:43:34.432072 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.432074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.432085 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.432116 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.432468 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55734: BlockingRecv error: recv got EOF from 127.0.0.1:55734 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 18:43:34.438890 1135 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:34.437870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.10.250.125:46803)
0620 18:43:34.437915 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.437917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.438183 (+ 266us) server_negotiation.cc:407] Connection header received
0620 18:43:34.438293 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.438294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.438307 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.438337 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.438830 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55742: BlockingRecv error: recv got EOF from 127.0.0.1:55742 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 18:43:34.449338 1135 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:34.448296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.10.250.125:46803)
0620 18:43:34.448356 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.448358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.448678 (+ 320us) server_negotiation.cc:407] Connection header received
0620 18:43:34.448794 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.448796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.448809 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.448841 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.449263 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55748: BlockingRecv error: recv got EOF from 127.0.0.1:55748 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 18:43:34.467896 1135 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:34.466868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.10.250.125:46803)
0620 18:43:34.466916 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.466918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.467236 (+ 318us) server_negotiation.cc:407] Connection header received
0620 18:43:34.467352 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.467353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.467367 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.467399 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.467831 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55764: BlockingRecv error: recv got EOF from 127.0.0.1:55764 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 18:43:34.502341 1135 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:34.501387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55780 (local address 127.10.250.125:46803)
0620 18:43:34.501436 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.501439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.501711 (+ 272us) server_negotiation.cc:407] Connection header received
0620 18:43:34.501816 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.501817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.501829 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.501860 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.502273 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55780: BlockingRecv error: recv got EOF from 127.0.0.1:55780 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 18:43:34.569049 1135 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:34.568148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55792 (local address 127.10.250.125:46803)
0620 18:43:34.568276 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.568279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.568391 (+ 112us) server_negotiation.cc:407] Connection header received
0620 18:43:34.568496 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.568497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.568510 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.568588 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.568971 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55792: BlockingRecv error: recv got EOF from 127.0.0.1:55792 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260620 18:43:34.699182 1135 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:34.698451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.10.250.125:46803)
0620 18:43:34.698575 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.698577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.698627 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:43:34.698702 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.698703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.698714 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.698780 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.699107 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55806: BlockingRecv error: recv got EOF from 127.0.0.1:55806 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 18:43:34.957231 1135 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:34.956557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.10.250.125:46803)
0620 18:43:34.956649 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:34.956651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:34.956662 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:43:34.956689 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:34.956689 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:34.956719 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:34.956808 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:34.957147 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55820: BlockingRecv error: recv got EOF from 127.0.0.1:55820 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:43:35.318215 1238 catalog_manager.cc:1259] Loaded cluster ID: bad4b00f0faa48c8aca6de4ae53fef6d
I20260620 18:43:35.318295 1238 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799: loading cluster ID for follower catalog manager: success
I20260620 18:43:35.319227 1238 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799: acquiring CA information for follower catalog manager: success
I20260620 18:43:35.319577 1238 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 67e602ef205a46e08b544c32fa308799: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:43:35.368393 1267 catalog_manager.cc:1259] Loaded cluster ID: bad4b00f0faa48c8aca6de4ae53fef6d
I20260620 18:43:35.368479 1267 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c: loading cluster ID for follower catalog manager: success
I20260620 18:43:35.369359 1267 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c: acquiring CA information for follower catalog manager: success
I20260620 18:43:35.369714 1267 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7806f1d33c4340ceacec95b30315821c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:43:35.473281 1339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:35.470625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55834 (local address 127.10.250.125:46803)
0620 18:43:35.470764 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:35.470766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:35.470778 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:43:35.470806 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:35.470807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:35.470859 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:35.470960 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:35.471137 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:35.471664 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:35.472470 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:35.472705 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:35.472767 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:35.472956 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:35.473074 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:35.473143 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:35.473180 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 18:43:35.473203 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260620 18:43:35.478302 1345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:35.475589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.10.250.124:41005)
0620 18:43:35.475725 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:35.475727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:35.475735 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:43:35.475775 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:35.475776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:35.475815 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:35.475893 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:35.476016 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:35.476544 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:35.477368 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:35.477613 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:35.477693 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:35.477875 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:35.478004 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:35.478116 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:35.478130 (+ 14us) server_negotiation.cc:299] Negotiation successful
0620 18:43:35.478150 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260620 18:43:35.483089 1352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:35.480236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.10.250.126:44271)
0620 18:43:35.480354 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:35.480356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:35.480501 (+ 145us) server_negotiation.cc:407] Connection header received
0620 18:43:35.480556 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:35.480557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:35.480610 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:35.480717 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:35.480850 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:35.481349 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:35.482133 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:35.482366 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:35.482454 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:35.482705 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:35.482842 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:35.482907 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:35.482964 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 18:43:35.482993 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260620 18:43:35.483515 1339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:35.480213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55836 (local address 127.10.250.125:46803)
0620 18:43:35.480266 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:35.480268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:35.480385 (+ 117us) server_negotiation.cc:407] Connection header received
0620 18:43:35.480505 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:35.480506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:35.480518 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:35.480548 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:35.480739 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:35.481563 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:35.482772 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:35.483065 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:35.483133 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:35.483275 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:35.483308 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:35.483359 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:35.483413 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 18:43:35.483437 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 18:43:35.484095 1345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:35.480555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.10.250.124:41005)
0620 18:43:35.480609 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:35.480611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:35.480618 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:43:35.480639 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:35.480640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:35.480650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:35.480698 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:35.480859 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:35.481561 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:35.483185 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:35.483487 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:35.483546 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:35.483837 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:35.483873 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:35.483918 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:35.483995 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 18:43:35.484019 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:43:35.485265 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 988
I20260620 18:43:35.491042 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 1063
I20260620 18:43:35.497869 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 1138
2026-06-20T18:43:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2873 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:35 dist-test-slave-wp6g krb5kdc[1356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:35 dist-test-slave-wp6g krb5kdc[1356](info): set up 2 sockets
Jun 20 18:43:35 dist-test-slave-wp6g krb5kdc[1356](info): commencing operation
krb5kdc: starting...
W20260620 18:43:36.563014 11241 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/test-user.keytab.
Jun 20 18:43:36 dist-test-slave-wp6g krb5kdc[1356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.250.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
Entry for principal kudu/127.10.250.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab.
I20260620 18:43:36.690945 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:36139
--webserver_interface=127.10.250.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:37449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:36139,127.10.250.125:33885,127.10.250.124:45169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:36.779597 1369 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:43:36.779893 1369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:36.779925 1369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:36.779970 1369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:36.781322 1369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:36.781411 1369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:36.781440 1369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:36.781461 1369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:36.781488 1369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:36.783247 1369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:37449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:36139,127.10.250.125:33885,127.10.250.124:45169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.126.keytab
--principal=kudu/127.10.250.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.126:36139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.250.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:36.783562 1369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:36.783829 1369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:36.789394 1374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:36.789603 1377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:36.789701 1375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:36.789899 1369 server_base.cc:1034] running on GCE node
I20260620 18:43:36.790189 1369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:36.790477 1369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:36.791713 1369 hybrid_clock.cc:648] HybridClock initialized: now 1781981016791688 us; error 40 us; skew 500 ppm
Jun 20 18:43:36 dist-test-slave-wp6g krb5kdc[1356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981016, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:36.800138 1369 init.cc:377] Logged in from keytab as kudu/127.10.250.126@KRBTEST.COM (short username kudu)
I20260620 18:43:36.800733 1369 webserver.cc:460] Webserver started at http://127.10.250.126:37835/ using document root <none> and password file <none>
I20260620 18:43:36.800973 1369 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:36.801020 1369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:36.801127 1369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:36.802254 1369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/data/instance:
uuid: "c5bc7fc00e9e435bbad7b80212f4fd93"
format_stamp: "Formatted at 2026-06-20 18:43:36 on dist-test-slave-wp6g"
server_key: "8f72b651b0db1484a5568c8a7c5ba28a"
server_key_iv: "68796781baa6d9636b5e53ed692f9795"
server_key_version: "encryptionkey@0"
I20260620 18:43:36.802685 1369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/wal/instance:
uuid: "c5bc7fc00e9e435bbad7b80212f4fd93"
format_stamp: "Formatted at 2026-06-20 18:43:36 on dist-test-slave-wp6g"
server_key: "8f72b651b0db1484a5568c8a7c5ba28a"
server_key_iv: "68796781baa6d9636b5e53ed692f9795"
server_key_version: "encryptionkey@0"
I20260620 18:43:36.804155 1369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:36.805094 1384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:36.805434 1369 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:36.805593 1369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/wal
uuid: "c5bc7fc00e9e435bbad7b80212f4fd93"
format_stamp: "Formatted at 2026-06-20 18:43:36 on dist-test-slave-wp6g"
server_key: "8f72b651b0db1484a5568c8a7c5ba28a"
server_key_iv: "68796781baa6d9636b5e53ed692f9795"
server_key_version: "encryptionkey@0"
I20260620 18:43:36.805691 1369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:36.815105 1369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:36.817541 1369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:36.817739 1369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:36.822173 1369 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.126:36139
I20260620 18:43:36.822209 1436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.126:36139 every 8 connection(s)
I20260620 18:43:36.822656 1369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:36.823575 1437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:36.825285 1437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:43:36.826622 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 1369
I20260620 18:43:36.826713 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-0/wal/instance
I20260620 18:43:36.826941 11241 external_mini_cluster.cc:1428] Setting key a5589c7b9af13eae8f7ca6a0567188a0
Jun 20 18:43:36 dist-test-slave-wp6g krb5kdc[1356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981016, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.250.125@KRBTEST.COM; defaulting to no policy
I20260620 18:43:36.837178 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:36.826052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:36139 (local address 127.0.0.1:49838)
0620 18:43:36.826243 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:36.826306 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:36.827539 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:36.827869 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:36.827872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:36.828311 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:36.828568 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:36.828575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:36.829964 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:36.829965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:36.830807 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:36.830810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:36.831342 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:36.831343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:36.831417 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:36.832026 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:36.832034 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:36.833505 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:36.835497 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:36.835503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:36.835506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:36.835713 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:36.836055 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:36.836056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:36.836056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:36.836151 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:36.836530 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:36.836537 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:36.836621 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 18:43:36.836764 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 18:43:36.836959 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":20,"thread_start_us":110,"threads_started":1}
I20260620 18:43:36.837183 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:36.826141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.10.250.126:36139)
0620 18:43:36.826326 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:36.826332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:36.826401 (+ 69us) server_negotiation.cc:407] Connection header received
0620 18:43:36.827730 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:36.827732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:36.827750 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:36.827823 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:36.828688 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:36.829873 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:36.830903 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:36.831239 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:36.831377 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:36.833599 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:36.833607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:36.833607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:36.833615 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:36.835402 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:36.835836 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:36.835839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:36.835839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:36.835936 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:36.836225 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:36.836226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:36.836227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:36.836448 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:36.836513 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:36.836794 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 18:43:36.837043 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.125@KRBTEST.COM" created.
I20260620 18:43:36.838081 1437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:43:36.838467 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:36.838333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:33885 (local address 127.0.0.1:50714)
0620 18:43:36.838391 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:36.838442 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 18:43:36.838599 1385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.250.125:33885: connect: Connection refused (error 111)
I20260620 18:43:36.839077 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:36.838973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:33885 (local address 127.0.0.1:50724)
0620 18:43:36.839024 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:36.839049 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 18:43:36.839188 1437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.125:33885: Network error: Client connection negotiation failed: client connection to 127.10.250.125:33885: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.125.keytab.
Entry for principal kudu/127.10.250.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.125.keytab.
I20260620 18:43:36.854796 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.125:33885
--webserver_interface=127.10.250.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.125.keytab
--principal=kudu/127.10.250.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:37449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:36139,127.10.250.125:33885,127.10.250.124:45169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:43:36.888442 1437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } attempt: 1
I20260620 18:43:36.889058 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:36.888891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:33885 (local address 127.0.0.1:50732)
0620 18:43:36.888965 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:36.889018 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 18:43:36.889447 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:36.889359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:33885 (local address 127.0.0.1:50742)
0620 18:43:36.889402 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:36.889430 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 18:43:36.889547 1437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.125:33885: Network error: Client connection negotiation failed: client connection to 127.10.250.125:33885: connect: Connection refused (error 111)
W20260620 18:43:36.939330 1444 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:43:36.939596 1444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:36.939616 1444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:36.939639 1444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:36.940876 1444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:36.940924 1444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:36.940938 1444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:36.940951 1444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:36.940963 1444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:36.942538 1444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:37449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:36139,127.10.250.125:33885,127.10.250.124:45169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.125.keytab
--principal=kudu/127.10.250.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.125:33885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.250.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:36.942824 1444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:36.943073 1444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:36.948444 1450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:36.948458 1444 server_base.cc:1034] running on GCE node
W20260620 18:43:36.948426 1452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:36.948426 1449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:36.948937 1444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:36.949260 1444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:36.950433 1444 hybrid_clock.cc:648] HybridClock initialized: now 1781981016950414 us; error 31 us; skew 500 ppm
Jun 20 18:43:36 dist-test-slave-wp6g krb5kdc[1356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981016, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:36.953197 1444 init.cc:377] Logged in from keytab as kudu/127.10.250.125@KRBTEST.COM (short username kudu)
I20260620 18:43:36.953629 1444 webserver.cc:460] Webserver started at http://127.10.250.125:34655/ using document root <none> and password file <none>
I20260620 18:43:36.953836 1444 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:36.953878 1444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:36.953971 1444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:36.955159 1444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/data/instance:
uuid: "f073893595584c03a7ce9dee69617c7a"
format_stamp: "Formatted at 2026-06-20 18:43:36 on dist-test-slave-wp6g"
server_key: "74d03aff43bf2c5bbf64249940c7c57f"
server_key_iv: "92b635c8fcedad7c8292e3ebe12c1a96"
server_key_version: "encryptionkey@0"
I20260620 18:43:36.955487 1444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/wal/instance:
uuid: "f073893595584c03a7ce9dee69617c7a"
format_stamp: "Formatted at 2026-06-20 18:43:36 on dist-test-slave-wp6g"
server_key: "74d03aff43bf2c5bbf64249940c7c57f"
server_key_iv: "92b635c8fcedad7c8292e3ebe12c1a96"
server_key_version: "encryptionkey@0"
I20260620 18:43:36.956892 1444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:43:36.957707 1459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:36.957690 1437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } attempt: 2
I20260620 18:43:36.958025 1444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:36.958135 1444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/wal
uuid: "f073893595584c03a7ce9dee69617c7a"
format_stamp: "Formatted at 2026-06-20 18:43:36 on dist-test-slave-wp6g"
server_key: "74d03aff43bf2c5bbf64249940c7c57f"
server_key_iv: "92b635c8fcedad7c8292e3ebe12c1a96"
server_key_version: "encryptionkey@0"
I20260620 18:43:36.958199 1444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:36.958220 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:36.958064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:33885 (local address 127.0.0.1:50746)
0620 18:43:36.958148 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:36.958188 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 18:43:36.958663 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:36.958488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:33885 (local address 127.0.0.1:50750)
0620 18:43:36.958595 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:36.958630 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.125:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260620 18:43:36.958760 1437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.125:33885: Network error: Client connection negotiation failed: client connection to 127.10.250.125:33885: connect: Connection refused (error 111)
I20260620 18:43:36.971750 1444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:36.974702 1444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:36.974846 1444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:36.979092 1444 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.125:33885
I20260620 18:43:36.979194 1511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.125:33885 every 8 connection(s)
I20260620 18:43:36.979575 1444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/data/info.pb
I20260620 18:43:36.980086 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 1444
I20260620 18:43:36.980172 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-1/wal/instance
I20260620 18:43:36.980391 11241 external_mini_cluster.cc:1428] Setting key 5efa10d569950671954e0eb36aedef55
I20260620 18:43:36.980867 1512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:36.982708 1512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.250.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.250.124@KRBTEST.COM" created.
Jun 20 18:43:36 dist-test-slave-wp6g krb5kdc[1356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981016, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.125@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:36.995494 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:36.983737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.10.250.126:36139)
0620 18:43:36.983814 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:36.983817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:36.984093 (+ 276us) server_negotiation.cc:407] Connection header received
0620 18:43:36.984691 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:36.984693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:36.984713 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:36.984786 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:36.985523 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:36.986040 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:36.987226 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:36.987466 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:36.987570 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:36.992021 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:36.992023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:36.992023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:36.992034 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:36.993803 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:36.994364 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:36.994366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:36.994367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:36.994437 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:36.994768 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:36.994769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:36.994769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:36.994914 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:36.994995 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:36.995298 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 18:43:36.995395 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:43:36.995900 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:36.983656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:36139 (local address 127.0.0.1:49852)
0620 18:43:36.983897 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:36.983984 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:36.984562 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:36.984872 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:36.984876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:36.985184 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:36.985405 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:36.985410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:36.986176 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:36.986178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:36.987109 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:36.987117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:36.987586 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:36.987587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:36.987767 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:36.988409 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:36.988431 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:36.991889 (+ 3458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:36.993915 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:36.993925 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:36.993928 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:36.994257 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:36.994541 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:36.994542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:36.994542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:36.994689 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:36.994987 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:36.994994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:36.995152 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 18:43:36.995264 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:43:36.995447 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":105,"threads_started":1}
I20260620 18:43:36.996639 1512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:36 dist-test-slave-wp6g krb5kdc[1356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981016, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.125@KRBTEST.COM for kudu/127.10.250.125@KRBTEST.COM
I20260620 18:43:37.006170 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:36.996932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:33885 (local address 127.0.0.1:50766)
0620 18:43:36.997047 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:36.997058 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:36.997094 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:36.997369 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:36.997370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:36.997543 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:36.997602 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:36.997604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:36.998503 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:36.998504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:36.999150 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:36.999153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:36.999718 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:36.999719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:36.999793 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.000356 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:37.000361 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:37.002464 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:37.004859 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:37.004860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:37.004861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:37.005104 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:37.005358 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:37.005359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:37.005359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:37.005419 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:37.005807 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:37.005808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:37.005865 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 18:43:37.005945 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 18:43:37.006042 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 18:43:37.006306 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:36.997019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.10.250.125:33885)
0620 18:43:36.997230 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:36.997234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:36.997254 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:43:36.997272 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:36.997274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:36.997285 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:36.997370 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:36.997684 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:36.998389 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:36.999265 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:36.999604 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:36.999718 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.002683 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:37.002695 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:37.002695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:37.002702 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:37.004734 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.005217 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.005221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.005222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.005289 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.005477 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.005477 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.005478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.005711 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:37.005784 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:37.005999 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 18:43:37.006139 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":182,"thread_start_us":115,"threads_started":1}
I20260620 18:43:37.007234 1512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:43:37.007664 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.007516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:45169 (local address 127.0.0.1:42040)
0620 18:43:37.007580 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:37.007637 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:45169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 18:43:37.007871 1462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.250.124:45169: connect: Connection refused (error 111)
I20260620 18:43:37.008441 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.008342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:45169 (local address 127.0.0.1:42052)
0620 18:43:37.008390 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:37.008418 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:45169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260620 18:43:37.008613 1512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.124:45169: Network error: Client connection negotiation failed: client connection to 127.10.250.124:45169: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.250.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.124.keytab.
Entry for principal kudu/127.10.250.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.124.keytab.
I20260620 18:43:37.010203 11241 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJBaTq/build/release/bin/kudu
/tmp/dist-test-taskGJBaTq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.124:45169
--webserver_interface=127.10.250.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.124.keytab
--principal=kudu/127.10.250.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.250.84:37449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.250.126:36139,127.10.250.125:33885,127.10.250.124:45169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:43:37.049929 1437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } attempt: 3
Jun 20 18:43:37 dist-test-slave-wp6g krb5kdc[1356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981016, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for kudu/127.10.250.125@KRBTEST.COM
I20260620 18:43:37.057765 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.050387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:33885 (local address 127.0.0.1:50776)
0620 18:43:37.050453 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:37.050472 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:37.050565 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:37.050729 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:37.050729 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:37.050910 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:37.050975 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:37.050977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.051692 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.051694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:37.052236 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:37.052238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.052657 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.052659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:37.052742 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.053258 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:37.053264 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:37.054858 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:37.056640 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:37.056641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:37.056641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:37.056836 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:37.057058 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:37.057059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:37.057059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:37.057126 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:37.057406 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:37.057408 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:37.057458 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 18:43:37.057543 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 18:43:37.057658 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 18:43:37.057765 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.050466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.10.250.125:33885)
0620 18:43:37.050548 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.050551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.050562 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:43:37.050637 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.050638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.050648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.050723 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.051066 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.051589 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.052327 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.052560 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.052652 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.054981 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:37.054983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:37.054983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:37.054998 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:37.056523 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.056916 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.056918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.056918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.056981 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.057197 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.057198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.057198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.057325 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:37.057401 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:37.057569 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 18:43:37.057664 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:43:37.057878 1512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } attempt: 1
I20260620 18:43:37.058332 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.058169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:45169 (local address 127.0.0.1:42062)
0620 18:43:37.058261 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:37.058293 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:45169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 18:43:37.058674 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.058585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:45169 (local address 127.0.0.1:42078)
0620 18:43:37.058623 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:37.058644 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:45169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 18:43:37.058638 1437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } has no permanent_uuid. Determining permanent_uuid...
W20260620 18:43:37.058771 1512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.124:45169: Network error: Client connection negotiation failed: client connection to 127.10.250.124:45169: connect: Connection refused (error 111)
I20260620 18:43:37.059011 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.058886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:45169 (local address 127.0.0.1:42092)
0620 18:43:37.058948 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:37.058983 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:45169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 18:43:37.059350 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.059268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:45169 (local address 127.0.0.1:42106)
0620 18:43:37.059314 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:37.059336 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:45169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 18:43:37.059453 1437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.124:45169: Network error: Client connection negotiation failed: client connection to 127.10.250.124:45169: connect: Connection refused (error 111)
I20260620 18:43:37.090705 1437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } attempt: 1
I20260620 18:43:37.091352 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.091156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:45169 (local address 127.0.0.1:42108)
0620 18:43:37.091257 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:37.091314 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:45169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 18:43:37.091859 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.091724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:45169 (local address 127.0.0.1:42110)
0620 18:43:37.091798 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:37.091834 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:45169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 18:43:37.092001 1437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.124:45169: Network error: Client connection negotiation failed: client connection to 127.10.250.124:45169: connect: Connection refused (error 111)
W20260620 18:43:37.097172 1519 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:43:37.097463 1519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:37.097514 1519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:37.097577 1519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:37.098982 1519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:37.099052 1519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:37.099139 1519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:37.099171 1519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:37.099195 1519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:37.100876 1519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.250.84:37449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.250.126:36139,127.10.250.125:33885,127.10.250.124:45169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/krb5kdc/kudu_127.10.250.124.keytab
--principal=kudu/127.10.250.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.250.124:45169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.250.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:43:37.101188 1519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:37.101447 1519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:37.106740 1527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:37.106815 1519 server_base.cc:1034] running on GCE node
W20260620 18:43:37.106747 1525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:37.106762 1524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:37.107240 1519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:37.107583 1519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:37.108811 1519 hybrid_clock.cc:648] HybridClock initialized: now 1781981017108790 us; error 39 us; skew 500 ppm
Jun 20 18:43:37 dist-test-slave-wp6g krb5kdc[1356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981017, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:37.111768 1519 init.cc:377] Logged in from keytab as kudu/127.10.250.124@KRBTEST.COM (short username kudu)
I20260620 18:43:37.112304 1519 webserver.cc:460] Webserver started at http://127.10.250.124:39045/ using document root <none> and password file <none>
I20260620 18:43:37.112511 1519 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:37.112563 1519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:37.112660 1519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:37.113732 1519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/data/instance:
uuid: "660dabb4930349e4839fb09593bb282a"
format_stamp: "Formatted at 2026-06-20 18:43:37 on dist-test-slave-wp6g"
server_key: "43bb5d57cb8bc7080718685b19433168"
server_key_iv: "4fe7335cbc0d746aaf6c53690eefde87"
server_key_version: "encryptionkey@0"
I20260620 18:43:37.114053 1519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/wal/instance:
uuid: "660dabb4930349e4839fb09593bb282a"
format_stamp: "Formatted at 2026-06-20 18:43:37 on dist-test-slave-wp6g"
server_key: "43bb5d57cb8bc7080718685b19433168"
server_key_iv: "4fe7335cbc0d746aaf6c53690eefde87"
server_key_version: "encryptionkey@0"
I20260620 18:43:37.115778 1519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:43:37.116541 1534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:37.116739 1519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:43:37.116832 1519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/data,/tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/wal
uuid: "660dabb4930349e4839fb09593bb282a"
format_stamp: "Formatted at 2026-06-20 18:43:37 on dist-test-slave-wp6g"
server_key: "43bb5d57cb8bc7080718685b19433168"
server_key_iv: "4fe7335cbc0d746aaf6c53690eefde87"
server_key_version: "encryptionkey@0"
I20260620 18:43:37.116894 1519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:37.126904 1512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } attempt: 2
I20260620 18:43:37.127404 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.127248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:45169 (local address 127.0.0.1:42112)
0620 18:43:37.127324 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:37.127372 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:45169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:43:37.127803 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.127723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:45169 (local address 127.0.0.1:42122)
0620 18:43:37.127762 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:37.127787 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.250.124:45169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 18:43:37.127907 1512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.250.124:45169: Network error: Client connection negotiation failed: client connection to 127.10.250.124:45169: connect: Connection refused (error 111)
I20260620 18:43:37.136804 1519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:37.139472 1519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:37.139649 1519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:37.143639 1519 rpc_server.cc:307] RPC server started. Bound to: 127.10.250.124:45169
I20260620 18:43:37.143679 1586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.250.124:45169 every 8 connection(s)
I20260620 18:43:37.144093 1519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/data/info.pb
I20260620 18:43:37.145287 1587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:37.145738 11241 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJBaTq/build/release/bin/kudu as pid 1519
I20260620 18:43:37.145849 11241 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJBaTq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980738687848-11241-0/minicluster-data/master-2/wal/instance
I20260620 18:43:37.146076 11241 external_mini_cluster.cc:1428] Setting key 6991777de1a1ed222d32427133691b42
I20260620 18:43:37.147671 1587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:37 dist-test-slave-wp6g krb5kdc[1356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981017, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.124@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
Jun 20 18:43:37 dist-test-slave-wp6g krb5kdc[1356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.124@KRBTEST.COM
I20260620 18:43:37.157595 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.148544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.10.250.126:36139)
0620 18:43:37.148598 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.148601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.148796 (+ 195us) server_negotiation.cc:407] Connection header received
0620 18:43:37.148840 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.148841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.148852 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.148905 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.149457 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.150031 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.151252 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.151550 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.151605 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.154151 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:37.154153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:37.154153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:37.154161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:37.155887 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.156446 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.156448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.156448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.156515 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.156815 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.156817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.156817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.156982 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:37.157051 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:37.157403 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 18:43:37.157495 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 18:43:37.157809 1590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.146821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42132 (local address 127.10.250.124:45169)
0620 18:43:37.147047 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.147056 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.147087 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:43:37.147533 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.147540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.147592 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.147721 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.148559 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.149660 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.150813 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.151288 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.151400 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.154705 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:37.154715 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:37.154715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:37.154723 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:37.156485 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.156873 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.156876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.156877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.156924 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.157126 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.157127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.157127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.157300 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:37.157460 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:37.157547 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 18:43:37.157615 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
I20260620 18:43:37.157974 1591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.148468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.126:36139 (local address 127.0.0.1:49862)
0620 18:43:37.148670 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:37.148717 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:37.148787 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:37.148974 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:37.148977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:37.149261 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:37.149361 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:37.149366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.150133 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.150134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:37.151152 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:37.151155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.151651 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.151652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:37.151740 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.152140 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:37.152159 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:37.154024 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:37.156031 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:37.156039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:37.156041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:37.156360 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:37.156632 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:37.156633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:37.156633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:37.156744 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:37.157055 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:37.157060 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:37.157251 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 18:43:37.157369 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 18:43:37.157547 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":105,"threads_started":1}
I20260620 18:43:37.158469 1587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:37 dist-test-slave-wp6g krb5kdc[1356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981017, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.124@KRBTEST.COM for kudu/127.10.250.125@KRBTEST.COM
Jun 20 18:43:37 dist-test-slave-wp6g krb5kdc[1356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.126@KRBTEST.COM
I20260620 18:43:37.165884 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.158843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50786 (local address 127.10.250.125:33885)
0620 18:43:37.158891 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.158893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.158900 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:43:37.158924 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.158925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.158935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.158989 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.159276 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.159759 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.160441 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.160658 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.160741 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.163042 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:37.163044 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:37.163044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:37.163052 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:37.164667 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.165054 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.165055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.165056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.165115 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.165336 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.165337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.165338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.165462 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:37.165535 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:37.165712 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 18:43:37.165783 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:43:37.165884 1591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.158762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.125:33885 (local address 127.0.0.1:50786)
0620 18:43:37.158800 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:37.158812 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:37.158851 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:37.158996 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:37.158997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:37.159143 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:37.159198 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:37.159199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.159841 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.159842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:37.160344 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:37.160346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.160741 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.160742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:37.160820 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.161425 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:37.161431 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:37.162952 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:37.164761 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:37.164762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:37.164763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:37.164955 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:37.165201 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:37.165202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:37.165202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:37.165263 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:37.165540 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:37.165541 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:37.165588 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 18:43:37.165674 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 18:43:37.165782 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 18:43:37.166262 1587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:43:37.166818 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.158900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.10.250.126:36139)
0620 18:43:37.158939 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.158941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.158989 (+ 48us) server_negotiation.cc:407] Connection header received
0620 18:43:37.159061 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.159062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.159071 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.159125 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.159600 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.160072 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.160777 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.161053 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.161125 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.163711 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:37.163713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:37.163713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:37.163720 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:37.165279 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.165693 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.165694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.165694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.165753 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.166190 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.166191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.166192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.166315 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:37.166559 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:37.166607 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 18:43:37.166694 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 18:43:37.167125 1437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } attempt: 2
Jun 20 18:43:37 dist-test-slave-wp6g krb5kdc[1356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981017, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.124@KRBTEST.COM for kudu/127.10.250.124@KRBTEST.COM
Jun 20 18:43:37 dist-test-slave-wp6g krb5kdc[1356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981016, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.126@KRBTEST.COM for kudu/127.10.250.124@KRBTEST.COM
Jun 20 18:43:37 dist-test-slave-wp6g krb5kdc[1356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.250.125@KRBTEST.COM
I20260620 18:43:37.173630 1591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.166608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:45169 (local address 127.0.0.1:42144)
0620 18:43:37.166640 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:37.166650 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:37.166690 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:37.166805 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:37.166806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:37.166958 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:37.166995 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:37.166997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.167886 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.167886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:37.168455 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:37.168458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.168840 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.168841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:37.168891 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.169354 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:37.169359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:37.170753 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:37.172371 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:37.172372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:37.172373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:37.172600 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:37.172823 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:37.172824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:37.172824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:37.172890 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:37.173256 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:37.173257 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:37.173296 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 18:43:37.173371 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 18:43:37.173496 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 18:43:37.173759 1590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.166664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42144 (local address 127.10.250.124:45169)
0620 18:43:37.166695 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.166697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.166704 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:43:37.166743 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.166744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.166753 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.166795 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.167071 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.167802 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.168539 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.168761 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.168832 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.170837 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:37.170838 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:37.170838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:37.170845 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:37.172290 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.172672 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.172673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.172673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.172746 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.172985 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.172986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.172986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.173144 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:37.173253 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:37.173408 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 18:43:37.173572 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":17}
I20260620 18:43:37.175400 1587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a: Bootstrap starting.
I20260620 18:43:37.176268 1587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:37.176625 1587 log.cc:824] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:37.176925 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.167420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.10.250.125:33885)
0620 18:43:37.167455 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.167456 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.167464 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:43:37.167517 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.167517 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.167526 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.167587 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.168180 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.168934 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.169875 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.170213 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.170288 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.173575 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:37.173576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:37.173576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:37.173583 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:37.175603 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.175939 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.175941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.175941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.176002 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.176204 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.176205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.176205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.176351 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:37.176520 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:37.176735 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 18:43:37.176819 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:43:37.177065 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.167345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:45169 (local address 127.0.0.1:42150)
0620 18:43:37.167419 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:37.167428 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:37.167460 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:37.167828 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:37.167829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:37.167986 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:37.168025 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:37.168027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.169326 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.169327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:37.170051 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:37.170053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.170566 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.170567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:37.170627 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.170971 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:37.170977 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:37.172992 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:37.175660 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:37.175661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:37.175661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:37.175918 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:37.176134 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:37.176135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:37.176135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:37.176210 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:37.176549 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:37.176551 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:37.176610 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 18:43:37.176704 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:43:37.176902 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:43:37.177618 1592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.167523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42150 (local address 127.10.250.124:45169)
0620 18:43:37.167681 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.167683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.167690 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:43:37.167709 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.167710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.167731 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.167804 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.168108 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.169233 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.170124 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.170376 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.170558 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.173067 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:37.173068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:37.173069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:37.173076 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:37.175538 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.176001 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.176002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.176002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.176058 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.176279 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.176280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.176280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.176441 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:37.176553 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:37.177386 (+ 833us) server_negotiation.cc:299] Negotiation successful
0620 18:43:37.177477 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260620 18:43:37.177989 1587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a: No bootstrap required, opened a new log
I20260620 18:43:37.178596 11241 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:37.179232 1437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93: Bootstrap starting.
I20260620 18:43:37.180011 1437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:37.179381 1411 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49878
I20260620 18:43:37.180155 1587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } }
I20260620 18:43:37.180336 1587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 660dabb4930349e4839fb09593bb282a, State: Initialized, Role: FOLLOWER
I20260620 18:43:37.180349 1437 log.cc:824] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:37.180429 1587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } }
I20260620 18:43:37.180734 1595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } } }
I20260620 18:43:37.180792 1587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:37.180850 1595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:37.181226 1437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93: No bootstrap required, opened a new log
I20260620 18:43:37.182596 1437 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } }
I20260620 18:43:37.182739 1437 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5bc7fc00e9e435bbad7b80212f4fd93, State: Initialized, Role: FOLLOWER
I20260620 18:43:37.182818 1437 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } }
I20260620 18:43:37.183167 1437 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:43:37.183183 1606 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 18:43:37.183200 1411 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
W20260620 18:43:37.183226 1606 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:43:37.183259 1411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:37.184055 1411 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } }
I20260620 18:43:37.184293 1411 leader_election.cc:288] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [CANDIDATE]: Term 1 election: Requested vote from peers f073893595584c03a7ce9dee69617c7a (127.10.250.125:33885), 660dabb4930349e4839fb09593bb282a (127.10.250.124:45169)
I20260620 18:43:37.182978 1519 master_runner.cc:186] Error getting master registration for 127.10.250.126:36139: OK, instance_id { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" instance_seqno: 1781981016820893 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:43:37.185385 1519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:43:37.186058 1561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "660dabb4930349e4839fb09593bb282a"
I20260620 18:43:37.186175 1561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:37.186612 1608 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } } }
I20260620 18:43:37.186691 1608 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:37.187170 1561 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5bc7fc00e9e435bbad7b80212f4fd93 in term 1.
I20260620 18:43:37.186851 1369 master_runner.cc:186] Error getting master registration for 127.10.250.125:33885: OK, instance_id { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" instance_seqno: 1781981016977656 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:43:37.189232 1369 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 18:43:37.185063 1619 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:43:37.189389 1619 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:43:37.189432 1387 leader_election.cc:302] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 660dabb4930349e4839fb09593bb282a, c5bc7fc00e9e435bbad7b80212f4fd93; no voters:
I20260620 18:43:37.189553 1608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:37.189750 1608 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [term 1 LEADER]: Becoming Leader. State: Replica: c5bc7fc00e9e435bbad7b80212f4fd93, State: Running, Role: LEADER
I20260620 18:43:37.189877 1608 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } }
I20260620 18:43:37.190354 1625 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5bc7fc00e9e435bbad7b80212f4fd93. Latest consensus state: current_term: 1 leader_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } } }
I20260620 18:43:37.190450 1625 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:37.200982 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.190863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.10.250.125:33885)
0620 18:43:37.190918 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.190921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.191287 (+ 366us) server_negotiation.cc:407] Connection header received
0620 18:43:37.191419 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.191420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.191431 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.191496 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.192311 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.193090 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.194092 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.194911 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.194972 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.197113 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:37.197115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:37.197115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:37.197125 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:37.199371 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.199839 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.199840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.199841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.199912 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.200145 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.200147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.200147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.200304 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:37.200379 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:37.200610 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 18:43:37.200710 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 18:43:37.202561 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.191097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.10.250.126:36139)
0620 18:43:37.191140 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.191143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.191679 (+ 536us) server_negotiation.cc:407] Connection header received
0620 18:43:37.191826 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.191827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.191852 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.191883 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.192970 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.193722 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.195269 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.195564 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.195608 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.197841 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:37.197842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:37.197843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:37.197853 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:37.199547 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.200392 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.200394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.200395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.200606 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.201719 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.201721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.201721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.201910 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:37.201979 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:37.202331 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 18:43:37.202424 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:43:37.209172 1592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.191293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.10.250.124:45169)
0620 18:43:37.191334 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.191336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.198687 (+ 7351us) server_negotiation.cc:407] Connection header received
0620 18:43:37.198798 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.198800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.198813 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.198845 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.201187 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.202000 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.203456 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.203825 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.203902 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.205529 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:37.205531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:37.205532 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:37.205540 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:37.207350 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.207792 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.207793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.207793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.207872 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.208074 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.208075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.208075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.208439 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:37.208585 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:37.208871 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 18:43:37.209041 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:43:37.210742 1627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:37.211014 1627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:37.213140 1561 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a [term 1 FOLLOWER]: Refusing update from remote peer c5bc7fc00e9e435bbad7b80212f4fd93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:37.213907 1625 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:37.216758 1595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5bc7fc00e9e435bbad7b80212f4fd93. Latest consensus state: current_term: 1 leader_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } } }
I20260620 18:43:37.216887 1595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:37.219039 1512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } attempt: 3
I20260620 18:43:37.219985 1608 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } } }
I20260620 18:43:37.220086 1608 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:37.220180 1631 mvcc.cc:205] Tried to move back new op lower bound from 7298994246501908480 to 7298994246410588160. Current Snapshot: MvccSnapshot[applied={T|T < 7298994246501908480}]
I20260620 18:43:37.220466 1595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } } }
I20260620 18:43:37.220573 1595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:37.223474 1632 mvcc.cc:205] Tried to move back new op lower bound from 7298994246501908480 to 7298994246410588160. Current Snapshot: MvccSnapshot[applied={T|T < 7298994246501908480}]
Jun 20 18:43:37 dist-test-slave-wp6g krb5kdc[1356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981016, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.250.125@KRBTEST.COM for kudu/127.10.250.124@KRBTEST.COM
I20260620 18:43:37.228816 1592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.219342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42158 (local address 127.10.250.124:45169)
0620 18:43:37.219509 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.219511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.219519 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:43:37.219576 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.219577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.219587 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.220740 (+ 1153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.221151 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.221900 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.222931 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.223252 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.223314 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.225519 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:37.225520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:37.225520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:37.225530 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:37.227539 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.227976 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.227977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.227977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.228054 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:37.228258 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:37.228259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:37.228260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:37.228403 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:37.228464 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:37.228637 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 18:43:37.228720 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 18:43:37.228816 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.219329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.250.124:45169 (local address 127.0.0.1:42158)
0620 18:43:37.219385 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:37.219403 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:37.219468 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:37.220762 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:37.220764 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:37.220959 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:37.221028 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:37.221031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.221981 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.221982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:37.222811 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:37.222814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.223319 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.223321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:37.223378 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.223904 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:37.223910 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:37.225412 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:37.227647 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:37.227648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:37.227648 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:37.227889 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:37.228111 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:37.228112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:37.228112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:37.228172 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:37.228463 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:37.228464 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:37.228512 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 18:43:37.228605 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:43:37.228720 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 18:43:37.230974 1512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a: Bootstrap starting.
I20260620 18:43:37.231734 1512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:37.232049 1512 log.cc:824] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:37.233108 1512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a: No bootstrap required, opened a new log
I20260620 18:43:37.235061 1627 catalog_manager.cc:1347] Generated new cluster ID: c903b3e0b63e4ee1a537303f41e6bab5
I20260620 18:43:37.235150 1627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:37.235617 1512 raft_consensus.cc:348] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } }
I20260620 18:43:37.235772 1512 raft_consensus.cc:729] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f073893595584c03a7ce9dee69617c7a, State: Initialized, Role: FOLLOWER
I20260620 18:43:37.235962 1512 consensus_queue.cc:260] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } }
I20260620 18:43:37.236425 1485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:37.236433 1486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f073893595584c03a7ce9dee69617c7a"
I20260620 18:43:37.236358 1512 sys_catalog.cc:564] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:37.237444 1485 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a [term 1 FOLLOWER]: Refusing update from remote peer c5bc7fc00e9e435bbad7b80212f4fd93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:37.237884 1608 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:37.239610 1486 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5bc7fc00e9e435bbad7b80212f4fd93 in term 1.
I20260620 18:43:37.236323 1637 sys_catalog.cc:455] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } } }
I20260620 18:43:37.242038 1637 sys_catalog.cc:458] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:37.242112 1637 sys_catalog.cc:455] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } } }
I20260620 18:43:37.242175 1637 sys_catalog.cc:458] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:37.242486 1649 sys_catalog.cc:455] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5bc7fc00e9e435bbad7b80212f4fd93. Latest consensus state: current_term: 1 leader_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } } }
I20260620 18:43:37.242645 1649 sys_catalog.cc:458] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:37.247107 1648 catalog_manager.cc:1259] Loaded cluster ID: c903b3e0b63e4ee1a537303f41e6bab5
I20260620 18:43:37.258584 1648 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a: loading cluster ID for follower catalog manager: success
W20260620 18:43:37.258939 1648 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260620 18:43:37.259065 1648 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:43:37.267789 1608 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bc7fc00e9e435bbad7b80212f4fd93" member_type: VOTER last_known_addr { host: "127.10.250.126" port: 36139 } } peers { permanent_uuid: "f073893595584c03a7ce9dee69617c7a" member_type: VOTER last_known_addr { host: "127.10.250.125" port: 33885 } } peers { permanent_uuid: "660dabb4930349e4839fb09593bb282a" member_type: VOTER last_known_addr { host: "127.10.250.124" port: 45169 } } }
I20260620 18:43:37.267922 1608 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:37.269291 1627 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:37.270134 1627 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:37.275820 1627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5bc7fc00e9e435bbad7b80212f4fd93: Generated new TSK 0
I20260620 18:43:37.276046 1627 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:43:37.300837 1517 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:37.299121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.10.250.125:33885)
0620 18:43:37.299166 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.299169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.300118 (+ 949us) server_negotiation.cc:407] Connection header received
0620 18:43:37.300236 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.300238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.300250 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.300283 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.300752 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50810: BlockingRecv error: recv got EOF from 127.0.0.1:50810 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 18:43:37.301767 1440 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:37.299537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.10.250.126:36139)
0620 18:43:37.299578 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.299580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.301263 (+ 1683us) server_negotiation.cc:407] Connection header received
0620 18:43:37.301360 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.301361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.301373 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.301402 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.301697 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49898: BlockingRecv error: recv got EOF from 127.0.0.1:49898 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 18:43:37.302665 1592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:37.299843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42168 (local address 127.10.250.124:45169)
0620 18:43:37.299886 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.299888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.302132 (+ 2244us) server_negotiation.cc:407] Connection header received
0620 18:43:37.302228 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.302229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.302240 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.302269 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.302602 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42168: BlockingRecv error: recv got EOF from 127.0.0.1:42168 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 18:43:37.305262 1517 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:37.304358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.10.250.125:33885)
0620 18:43:37.304394 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.304395 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.304636 (+ 241us) server_negotiation.cc:407] Connection header received
0620 18:43:37.304745 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.304746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.304766 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.304795 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.305212 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50826: BlockingRecv error: recv got EOF from 127.0.0.1:50826 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 18:43:37.308486 1517 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:37.307569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.10.250.125:33885)
0620 18:43:37.307604 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.307605 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.307897 (+ 292us) server_negotiation.cc:407] Connection header received
0620 18:43:37.308073 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.308074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.308084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.308113 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.308436 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50842: BlockingRecv error: recv got EOF from 127.0.0.1:50842 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 18:43:37.312495 1517 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:37.311661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.10.250.125:33885)
0620 18:43:37.311697 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.311699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.311976 (+ 277us) server_negotiation.cc:407] Connection header received
0620 18:43:37.312088 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.312089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.312100 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.312136 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.312446 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50846: BlockingRecv error: recv got EOF from 127.0.0.1:50846 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 18:43:37.318944 1517 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:37.317920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.10.250.125:33885)
0620 18:43:37.317966 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.317969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.318301 (+ 332us) server_negotiation.cc:407] Connection header received
0620 18:43:37.318407 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.318409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.318420 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.318451 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.318880 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50858: BlockingRecv error: recv got EOF from 127.0.0.1:50858 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 18:43:37.329463 1517 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:37.328567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.10.250.125:33885)
0620 18:43:37.328611 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.328614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.328859 (+ 245us) server_negotiation.cc:407] Connection header received
0620 18:43:37.328964 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.328966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.328977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.329019 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.329401 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50874: BlockingRecv error: recv got EOF from 127.0.0.1:50874 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 18:43:37.348122 1517 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:37.347129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50876 (local address 127.10.250.125:33885)
0620 18:43:37.347175 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.347178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.347455 (+ 277us) server_negotiation.cc:407] Connection header received
0620 18:43:37.347567 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.347568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.347581 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.347614 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.348056 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50876: BlockingRecv error: recv got EOF from 127.0.0.1:50876 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 18:43:37.383026 1517 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:37.381990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.10.250.125:33885)
0620 18:43:37.382038 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.382041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.382331 (+ 290us) server_negotiation.cc:407] Connection header received
0620 18:43:37.382442 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.382444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.382469 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.382503 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.382964 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50884: BlockingRecv error: recv got EOF from 127.0.0.1:50884 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 18:43:37.449702 1517 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:37.448821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.10.250.125:33885)
0620 18:43:37.448970 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.448972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.449028 (+ 56us) server_negotiation.cc:407] Connection header received
0620 18:43:37.449114 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.449115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.449126 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.449189 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.449641 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50892: BlockingRecv error: recv got EOF from 127.0.0.1:50892 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260620 18:43:37.580463 1517 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:37.579182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50898 (local address 127.10.250.125:33885)
0620 18:43:37.579366 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.579368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.579755 (+ 387us) server_negotiation.cc:407] Connection header received
0620 18:43:37.579792 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.579793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.579806 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.579872 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.580391 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50898: BlockingRecv error: recv got EOF from 127.0.0.1:50898 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
W20260620 18:43:37.838418 1517 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:37.837688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50908 (local address 127.10.250.125:33885)
0620 18:43:37.837798 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.837800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.837812 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:43:37.837901 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.837902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.837913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.837966 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.838353 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50908: BlockingRecv error: recv got EOF from 127.0.0.1:50908 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:43:38.183799 1606 catalog_manager.cc:1259] Loaded cluster ID: c903b3e0b63e4ee1a537303f41e6bab5
I20260620 18:43:38.183885 1606 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a: loading cluster ID for follower catalog manager: success
I20260620 18:43:38.184835 1606 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a: acquiring CA information for follower catalog manager: success
I20260620 18:43:38.185158 1606 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 660dabb4930349e4839fb09593bb282a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:43:38.260226 1648 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a: acquiring CA information for follower catalog manager: success
I20260620 18:43:38.260654 1648 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f073893595584c03a7ce9dee69617c7a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:43:38.354597 1721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:38.351733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50910 (local address 127.10.250.125:33885)
0620 18:43:38.351895 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:38.351897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:38.351908 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:43:38.351976 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:38.351977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:38.352030 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:38.352113 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:38.352278 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:38.352792 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:38.353621 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:38.353823 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:38.353895 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:38.354140 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:38.354268 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:38.354331 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:38.354419 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 18:43:38.354443 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
I20260620 18:43:38.359112 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:38.356478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.10.250.124:45169)
0620 18:43:38.356582 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:38.356584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:38.356591 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:43:38.356633 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:38.356634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:38.356669 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:38.356739 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:38.356877 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:38.357389 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:38.358144 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:38.358348 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:38.358411 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:38.358684 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:38.358812 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:38.358887 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:38.358963 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 18:43:38.358989 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":56,"threads_started":1}
I20260620 18:43:38.364162 1721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:38.361066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50922 (local address 127.10.250.125:33885)
0620 18:43:38.361134 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:38.361136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:38.361278 (+ 142us) server_negotiation.cc:407] Connection header received
0620 18:43:38.361389 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:38.361390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:38.361402 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:38.361433 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:38.361681 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:38.362493 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:38.363298 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:38.363599 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:38.363663 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:38.363831 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:38.363868 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:38.363932 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:38.364038 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 18:43:38.364061 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 18:43:38.364292 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:38.361344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.10.250.124:45169)
0620 18:43:38.361386 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:38.361387 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:38.361555 (+ 168us) server_negotiation.cc:407] Connection header received
0620 18:43:38.361601 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:38.361602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:38.361613 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:38.361675 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:38.361785 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:38.362499 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:38.363518 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:38.363794 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:38.363851 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:38.364020 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:38.364051 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:38.364077 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:38.364190 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 18:43:38.364215 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:43:38.364429 1734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:38.361098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49902 (local address 127.10.250.126:36139)
0620 18:43:38.361199 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:38.361201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:38.361209 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:43:38.361292 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:38.361292 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:38.361343 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:38.361425 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:38.361611 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:38.362311 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:38.363397 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:38.363707 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:38.363772 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:38.363966 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:38.364086 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:38.364147 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:38.364306 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 18:43:38.364336 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":48,"threads_started":1}
I20260620 18:43:38.366668 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 1369
I20260620 18:43:38.372367 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 1444
I20260620 18:43:38.377511 11241 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJBaTq/build/release/bin/kudu with pid 1519
2026-06-20T18:43:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2879 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5754 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (279698 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST